Machine Learning Engineer Nanodegree

Capstone

Project- Corn Commodity Futures Price Predictor

In this project we will try to predict closing weekly price of Corn Commodity Futures. In order to perform this prediction we will create a dataset that includes weekly Corn Futures closing prices as well as Long Open Interest and Short Open Interest of Processors/Users( sometimes they are called Commercials) from COT reports and by using this dataset we will try to predict next week’s prices.

1. Data Sets

Historical Futures Prices: Corn Futures, Continuous Contract #1. Non-adjusted price based on spot-month continuous contract calculations. Raw data from CME: Can be found here
Commitment of Traders - CORN (CBT) - Futures Only (002602) Can be found here

Data has been downloaded and stored in \Data folder:

  • .\data\CHRIS-CME_C1.csv - Corn Futures Prices data
  • .\data\CFTC-002602_F_ALL.csv - Commitment of Traders data
In [1]:
import warnings
warnings.filterwarnings('ignore')
In [2]:
import pandas as pd
import numpy as np
from IPython.core.display import display, HTML
pd.options.display.max_colwidth = 500  # You need this, otherwise pandas
# will limit your HTML strings to 50 characters
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
pd.options.mode.chained_assignment = None  # default='warn'
from matplotlib import pyplot
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from math import sqrt
from numpy import concatenate
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
import cufflinks as cf
import plotly.tools as tls
init_notebook_mode(connected=True)
cf.go_offline()
Using TensorFlow backend.
C:\Users\zilvi\Anaconda3\envs\zil_tensorflow\lib\site-packages\plotly\graph_objs\_deprecations.py:558: DeprecationWarning:

plotly.graph_objs.YAxis is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.layout.YAxis
  - plotly.graph_objs.layout.scene.YAxis


C:\Users\zilvi\Anaconda3\envs\zil_tensorflow\lib\site-packages\plotly\graph_objs\_deprecations.py:531: DeprecationWarning:

plotly.graph_objs.XAxis is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.layout.XAxis
  - plotly.graph_objs.layout.scene.XAxis


2. Prepare and Explore Data

In [3]:
df_fut_orig = pd.read_csv('data\CHRIS-CME_C1.csv')
df_fut_orig.head(n=5)
Out[3]:
Date Open High Low Last Change Settle Volume Previous_Day_Open_Interest
0 2018-07-10 344.25 344.75 336.25 339.50 6.00 339.75 2668.0 2186.0
1 2018-07-09 346.00 348.50 342.50 346.00 6.00 345.75 3190.0 2969.0
2 2018-07-06 342.00 352.25 342.00 350.75 8.25 351.75 3068.0 3959.0
3 2018-07-05 345.50 348.75 341.50 342.50 0.75 343.50 3302.0 4812.0
4 2018-07-03 340.25 345.25 339.25 343.25 5.25 342.75 3048.0 5687.0
In [4]:
# Display a description of the dataset
display(df_fut_orig.describe())
Open High Low Last Change Settle Volume Previous_Day_Open_Interest
count 3033.000000 3034.000000 3034.000000 3034.000000 1081.000000 3034.000000 3034.000000 3034.00000
mean 457.095038 462.322924 451.795485 456.920040 3.950324 456.979318 103905.200396 352140.90145
std 140.338892 142.056030 138.436196 140.243019 3.415126 140.204571 73993.219920 248565.85531
min 219.000000 220.750000 216.750000 219.000000 0.000000 219.000000 0.000000 107.00000
25% 360.000000 363.000000 356.250000 359.500000 1.500000 359.750000 40172.750000 107559.25000
50% 388.500000 392.000000 383.500000 388.750000 3.000000 389.000000 102567.000000 365073.00000
75% 565.500000 573.562500 557.375000 564.625000 5.500000 564.625000 152391.250000 556408.50000
max 830.250000 843.750000 822.750000 831.250000 30.750000 831.250000 538170.000000 858696.00000
In [5]:
df_fut_orig['Date'] = pd.to_datetime(df_fut_orig['Date'])
df_fut_orig.set_index('Date',inplace=True)
df_fut_orig = df_fut_orig.sort_values('Date')

Plot Corn Futures Price Series using Plotly

In [6]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_original_price_series(df_fut_orig)

Seems there are some rows where Volume=0, lets find out more about these rows

In [7]:
df_fut_orig[df_fut_orig['Volume']<1]
Out[7]:
Open High Low Last Change Settle Volume Previous_Day_Open_Interest
Date
2007-04-05 359.75 367.50 357.25 366.00 NaN 366.00 0.0 354349.0
2012-04-06 658.25 658.25 658.25 658.25 NaN 658.25 0.0 401521.0
2015-04-03 386.50 386.50 386.50 386.50 NaN 386.50 0.0 470964.0

Since we will resample daily prices into weekly prices , lets drop those rows.

In [8]:
# drop outliers
df_fut_orig.drop(df_fut_orig[df_fut_orig.Volume<1].index, inplace=True)
In [9]:
df_cot_orig = pd.read_csv('data\CFTC-002602_F_ALL.csv')
display(df_cot_orig.head())
Date Open_Interest Producer_Merchant_Processor_User_Longs Producer_Merchant_Processor_User_Shorts Swap Dealer Longs Swap Dealer Shorts Swap Dealer Spreads Money Manager Longs Money Manager Shorts Money Manager Spreads Other Reportable Longs Other Reportable Shorts Other Reportable Spreads Total Reportable Longs Total Reportable Shorts Non Reportable Longs Non Reportable Shorts
0 2018-07-10 1818055.0 500172.0 750062.0 208128.0 39513.0 99477.0 263353.0 404297.0 154286.0 320946.0 70682.0 98709.0 1645071.0 1617026.0 172984.0 201029.0
1 2018-07-03 1830330.0 484257.0 773851.0 210341.0 36927.0 100340.0 274795.0 382191.0 149756.0 322256.0 66508.0 119627.0 1661372.0 1629200.0 168958.0 201130.0
2 2018-06-26 1885804.0 513100.0 840177.0 223131.0 32763.0 91972.0 287061.0 377825.0 153461.0 330396.0 58283.0 116745.0 1715866.0 1671226.0 169938.0 214578.0
3 2018-06-19 1992169.0 525197.0 920764.0 222105.0 41144.0 99285.0 299377.0 356828.0 163454.0 379025.0 56652.0 135078.0 1823521.0 1773205.0 168648.0 218964.0
4 2018-06-12 1963233.0 488666.0 917204.0 235249.0 37674.0 93281.0 292054.0 304292.0 172623.0 363918.0 65030.0 147098.0 1792889.0 1737202.0 170344.0 226031.0
In [10]:
display(df_cot_orig.describe())
Open_Interest Producer_Merchant_Processor_User_Longs Producer_Merchant_Processor_User_Shorts Swap Dealer Longs Swap Dealer Shorts Swap Dealer Spreads Money Manager Longs Money Manager Shorts Money Manager Spreads Other Reportable Longs Other Reportable Shorts Other Reportable Spreads Total Reportable Longs Total Reportable Shorts Non Reportable Longs Non Reportable Shorts
count 6.310000e+02 631.000000 6.310000e+02 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 6.310000e+02 6.310000e+02 631.000000 631.000000
mean 1.292201e+06 270795.049128 6.268425e+05 290792.497623 20337.034865 33260.068146 236884.269414 137472.426307 94546.356577 140931.890650 70914.334390 85505.109350 1.152715e+06 1.068878e+06 139485.541997 223322.976228
std 2.095471e+05 68976.221600 1.554272e+05 53203.484072 18944.008732 22912.567257 67454.195123 109465.025186 32739.133163 51939.690903 26360.863384 29682.425476 1.939790e+05 2.060080e+05 23718.957966 29824.710288
min 7.482520e+05 102373.000000 2.972960e+05 186981.000000 0.000000 4397.000000 96989.000000 6714.000000 29130.000000 49809.000000 25905.000000 27592.000000 6.379810e+05 5.689510e+05 78578.000000 156086.000000
25% 1.192226e+06 226595.000000 5.235930e+05 255196.500000 6524.000000 13978.000000 186366.500000 47947.000000 72018.500000 104764.000000 53331.000000 62690.000000 1.055362e+06 9.573815e+05 121829.500000 198860.500000
50% 1.301506e+06 262823.000000 6.112810e+05 276337.000000 15239.000000 27209.000000 225682.000000 95548.000000 91850.000000 140343.000000 66261.000000 82705.000000 1.166372e+06 1.067548e+06 136966.000000 227337.000000
75% 1.398275e+06 314224.000000 7.058555e+05 321265.500000 28178.000000 48009.500000 287331.000000 211154.000000 113803.000000 175846.000000 83448.500000 106077.500000 1.247976e+06 1.180280e+06 153542.500000 246903.000000
max 1.992169e+06 525197.000000 1.001517e+06 422803.000000 95591.000000 113775.000000 431569.000000 447470.000000 231064.000000 379025.000000 173322.000000 181385.000000 1.825238e+06 1.773205e+06 206821.000000 293948.000000

Drop unnecessary columns columns and resample data

In [11]:
df_fut=df_fut_orig.drop(columns=[clmn for i,clmn in enumerate(df_fut_orig.columns) if i not in [5,6,7] ],axis=1)
display(df_fut.head())
Settle Volume Previous_Day_Open_Interest
Date
2006-06-16 235.50 56486.0 203491.0
2006-06-19 229.75 51299.0 190044.0
2006-06-20 229.75 41605.0 175859.0
2006-06-21 232.75 29803.0 162348.0
2006-06-22 230.50 28687.0 147658.0
In [12]:
s_settle =df_fut['Settle'].resample('W').last()
s_volume =df_fut['Volume'].resample('W').last()
df_fut_weekly = pd.concat([s_settle,s_volume], axis=1)
display(df_fut_weekly.head())
Settle Volume
Date
2006-06-18 235.50 56486.0
2006-06-25 228.25 28361.0
2006-07-02 235.50 30519.0
2006-07-09 241.00 13057.0
2006-07-16 253.50 2460.0
In [13]:
df_cot=df_cot_orig.drop(columns=[clmn for i,clmn in enumerate(df_cot_orig.columns) if i not in [0,1,2,3 ]],axis=1)
df_cot.rename(index=str, columns={"Producer_Merchant_Processor_User_Longs": "Longs", \
                                  "Producer_Merchant_Processor_User_Shorts": "Shorts"},inplace=True)
df_cot['Date'] = pd.to_datetime(df_cot['Date'])
df_cot.set_index('Date',inplace=True)
display(df_cot.head())
Open_Interest Longs Shorts
Date
2018-07-10 1818055.0 500172.0 750062.0
2018-07-03 1830330.0 484257.0 773851.0
2018-06-26 1885804.0 513100.0 840177.0
2018-06-19 1992169.0 525197.0 920764.0
2018-06-12 1963233.0 488666.0 917204.0
In [14]:
s_longs =df_cot['Longs'].resample('W').last()
s_shorts =df_cot['Shorts'].resample('W').last()
s_open_interest =df_cot['Open_Interest'].resample('W').last()
df_cot_weekly = pd.concat([s_open_interest,s_longs, s_shorts], axis=1)
display(df_cot_weekly.head(5))
Open_Interest Longs Shorts
Date
2006-06-18 1320155.0 209662.0 699163.0
2006-06-25 1321520.0 224476.0 666688.0
2006-07-02 1329400.0 234769.0 645735.0
2006-07-09 1327482.0 220552.0 648405.0
2006-07-16 1333225.0 216968.0 673110.0
In [15]:
df_weekly = pd.merge(df_fut_weekly,df_cot_weekly, on='Date')
display(df_weekly.head(5))
Settle Volume Open_Interest Longs Shorts
Date
2006-06-18 235.50 56486.0 1320155.0 209662.0 699163.0
2006-06-25 228.25 28361.0 1321520.0 224476.0 666688.0
2006-07-02 235.50 30519.0 1329400.0 234769.0 645735.0
2006-07-09 241.00 13057.0 1327482.0 220552.0 648405.0
2006-07-16 253.50 2460.0 1333225.0 216968.0 673110.0
In [16]:
# Display a description of the dataset
display(df_weekly.describe())
Settle Volume Open_Interest Longs Shorts
count 631.000000 631.000000 6.310000e+02 631.000000 6.310000e+02
mean 456.978605 100835.204437 1.292201e+06 270795.049128 6.268425e+05
std 140.242112 72466.341538 2.095471e+05 68976.221600 1.554272e+05
min 219.750000 132.000000 7.482520e+05 102373.000000 2.972960e+05
25% 359.500000 34822.500000 1.192226e+06 226595.000000 5.235930e+05
50% 389.250000 101209.000000 1.301506e+06 262823.000000 6.112810e+05
75% 560.375000 150341.000000 1.398275e+06 314224.000000 7.058555e+05
max 824.500000 369522.000000 1.992169e+06 525197.000000 1.001517e+06
In [17]:
# rest index since we need row numbers for splitting
df_weekly_idx_date=df_weekly.copy()
df_weekly.reset_index(inplace=True)

3. Visualise Data

In [18]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_weekly_combined_series_by_date(df_weekly)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [19]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_weekly_combined_series_by_trading_week(df_weekly)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [20]:
%load_ext autoreload
%autoreload 2
import visuals
visuals.plot_grouped_by_year_data(df_weekly_idx_date,"Stacked Plots of Price by Year")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [21]:
%load_ext autoreload
%autoreload 2
import visuals
visuals.lag_plot(df_weekly,"Lag Plot")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

4. Normalise the data using minmaxscaler

In [22]:
scaler = MinMaxScaler(feature_range=(0, 1))
values = df_weekly.loc[:, df_weekly.columns != 'Date'].values
scaled = scaler.fit_transform(values)

5. Split data into training, validation and test sets

In [23]:
validation_start=df_weekly[df_weekly['Date'] >= pd.to_datetime('2017-01-01')].index[0]
testing_start=df_weekly[df_weekly['Date'] >= pd.to_datetime('2018-01-01')].index[0]
In [24]:
print("validation start",validation_start)
print("testing start",testing_start)
validation start 550
testing start 603
In [25]:
# print data to double check
#print(df_weekly.iloc[validation_start])
#print(df_weekly.iloc[testing_start])
In [26]:
%load_ext autoreload
%autoreload 2
import data_preparer
reframed = data_preparer.series_to_supervised(scaled, 1, 1)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [27]:
# drop columns we don't want to predict
reframed.drop(reframed.columns[[6,7,8,9]], axis=1, inplace=True)
In [28]:
display(reframed.head())
var1(t-1) var2(t-1) var3(t-1) var4(t-1) var5(t-1) var1(t)
1 0.026044 0.152560 0.459760 0.253744 0.570655 0.014055
2 0.014055 0.076421 0.460857 0.288780 0.524540 0.026044
3 0.026044 0.082263 0.467192 0.313123 0.494786 0.035138
4 0.035138 0.034990 0.465650 0.279499 0.498578 0.055808
5 0.055808 0.006302 0.470267 0.271023 0.533659 0.028938

6. Define and Fit Model

In [66]:
%load_ext autoreload
%autoreload 2
import data_preparer
train_X, train_y, validation_X, validation_y,test_X, test_y = data_preparer.split_data(reframed,validation_start,testing_start)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [67]:
%load_ext autoreload
%autoreload 2
import models
model,history=models.basic_lstm_model(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Train on 550 samples, validate on 53 samples
Epoch 1/500
 - 14s - loss: 0.5592 - val_loss: 0.3919
Epoch 2/500
 - 0s - loss: 0.5173 - val_loss: 0.3437
Epoch 3/500
 - 0s - loss: 0.4764 - val_loss: 0.2974
Epoch 4/500
 - 0s - loss: 0.4374 - val_loss: 0.2533
Epoch 5/500
 - 0s - loss: 0.4009 - val_loss: 0.2120
Epoch 6/500
 - 0s - loss: 0.3678 - val_loss: 0.1729
Epoch 7/500
 - 0s - loss: 0.3363 - val_loss: 0.1351
Epoch 8/500
 - 0s - loss: 0.3058 - val_loss: 0.0985
Epoch 9/500
 - 0s - loss: 0.2770 - val_loss: 0.0638
Epoch 10/500
 - 0s - loss: 0.2515 - val_loss: 0.0362
Epoch 11/500
 - 0s - loss: 0.2331 - val_loss: 0.0267
Epoch 12/500
 - 0s - loss: 0.2226 - val_loss: 0.0308
Epoch 13/500
 - 0s - loss: 0.2165 - val_loss: 0.0382
Epoch 14/500
 - 0s - loss: 0.2126 - val_loss: 0.0453
Epoch 15/500
 - 0s - loss: 0.2100 - val_loss: 0.0516
Epoch 16/500
 - 0s - loss: 0.2083 - val_loss: 0.0560
Epoch 17/500
 - 0s - loss: 0.2071 - val_loss: 0.0595
Epoch 18/500
 - 0s - loss: 0.2062 - val_loss: 0.0622
Epoch 19/500
 - 0s - loss: 0.2054 - val_loss: 0.0641
Epoch 20/500
 - 0s - loss: 0.2047 - val_loss: 0.0655
Epoch 21/500
 - 0s - loss: 0.2041 - val_loss: 0.0665
Epoch 22/500
 - 0s - loss: 0.2036 - val_loss: 0.0669
Epoch 23/500
 - 0s - loss: 0.2031 - val_loss: 0.0670
Epoch 24/500
 - 0s - loss: 0.2026 - val_loss: 0.0672
Epoch 25/500
 - 0s - loss: 0.2021 - val_loss: 0.0674
Epoch 26/500
 - 0s - loss: 0.2016 - val_loss: 0.0675
Epoch 27/500
 - 0s - loss: 0.2012 - val_loss: 0.0676
Epoch 28/500
 - 0s - loss: 0.2007 - val_loss: 0.0677
Epoch 29/500
 - 0s - loss: 0.2002 - val_loss: 0.0678
Epoch 30/500
 - 0s - loss: 0.1998 - val_loss: 0.0677
Epoch 31/500
 - 0s - loss: 0.1994 - val_loss: 0.0675
Epoch 32/500
 - 0s - loss: 0.1989 - val_loss: 0.0672
Epoch 33/500
 - 0s - loss: 0.1985 - val_loss: 0.0669
Epoch 34/500
 - 0s - loss: 0.1980 - val_loss: 0.0665
Epoch 35/500
 - 0s - loss: 0.1976 - val_loss: 0.0662
Epoch 36/500
 - 0s - loss: 0.1972 - val_loss: 0.0659
Epoch 37/500
 - 0s - loss: 0.1967 - val_loss: 0.0655
Epoch 38/500
 - 0s - loss: 0.1963 - val_loss: 0.0652
Epoch 39/500
 - 0s - loss: 0.1959 - val_loss: 0.0648
Epoch 40/500
 - 0s - loss: 0.1954 - val_loss: 0.0644
Epoch 41/500
 - 0s - loss: 0.1950 - val_loss: 0.0639
Epoch 42/500
 - 0s - loss: 0.1945 - val_loss: 0.0634
Epoch 43/500
 - 0s - loss: 0.1941 - val_loss: 0.0630
Epoch 44/500
 - 0s - loss: 0.1936 - val_loss: 0.0626
Epoch 45/500
 - 0s - loss: 0.1931 - val_loss: 0.0621
Epoch 46/500
 - 0s - loss: 0.1927 - val_loss: 0.0616
Epoch 47/500
 - 0s - loss: 0.1922 - val_loss: 0.0612
Epoch 48/500
 - 0s - loss: 0.1917 - val_loss: 0.0608
Epoch 49/500
 - 0s - loss: 0.1912 - val_loss: 0.0604
Epoch 50/500
 - 0s - loss: 0.1907 - val_loss: 0.0601
Epoch 51/500
 - 0s - loss: 0.1902 - val_loss: 0.0598
Epoch 52/500
 - 0s - loss: 0.1896 - val_loss: 0.0595
Epoch 53/500
 - 0s - loss: 0.1891 - val_loss: 0.0592
Epoch 54/500
 - 0s - loss: 0.1885 - val_loss: 0.0589
Epoch 55/500
 - 0s - loss: 0.1879 - val_loss: 0.0587
Epoch 56/500
 - 0s - loss: 0.1873 - val_loss: 0.0585
Epoch 57/500
 - 0s - loss: 0.1867 - val_loss: 0.0583
Epoch 58/500
 - 0s - loss: 0.1860 - val_loss: 0.0582
Epoch 59/500
 - 0s - loss: 0.1854 - val_loss: 0.0582
Epoch 60/500
 - 0s - loss: 0.1847 - val_loss: 0.0582
Epoch 61/500
 - 0s - loss: 0.1840 - val_loss: 0.0582
Epoch 62/500
 - 0s - loss: 0.1832 - val_loss: 0.0580
Epoch 63/500
 - 0s - loss: 0.1825 - val_loss: 0.0578
Epoch 64/500
 - 0s - loss: 0.1817 - val_loss: 0.0577
Epoch 65/500
 - 0s - loss: 0.1808 - val_loss: 0.0577
Epoch 66/500
 - 0s - loss: 0.1800 - val_loss: 0.0576
Epoch 67/500
 - 0s - loss: 0.1791 - val_loss: 0.0574
Epoch 68/500
 - 0s - loss: 0.1781 - val_loss: 0.0572
Epoch 69/500
 - 0s - loss: 0.1772 - val_loss: 0.0570
Epoch 70/500
 - 0s - loss: 0.1762 - val_loss: 0.0565
Epoch 71/500
 - 0s - loss: 0.1752 - val_loss: 0.0560
Epoch 72/500
 - 0s - loss: 0.1741 - val_loss: 0.0552
Epoch 73/500
 - 0s - loss: 0.1730 - val_loss: 0.0546
Epoch 74/500
 - 0s - loss: 0.1719 - val_loss: 0.0541
Epoch 75/500
 - 0s - loss: 0.1707 - val_loss: 0.0534
Epoch 76/500
 - 0s - loss: 0.1695 - val_loss: 0.0527
Epoch 77/500
 - 0s - loss: 0.1683 - val_loss: 0.0522
Epoch 78/500
 - 0s - loss: 0.1670 - val_loss: 0.0518
Epoch 79/500
 - 0s - loss: 0.1656 - val_loss: 0.0515
Epoch 80/500
 - 0s - loss: 0.1642 - val_loss: 0.0510
Epoch 81/500
 - 0s - loss: 0.1628 - val_loss: 0.0505
Epoch 82/500
 - 0s - loss: 0.1613 - val_loss: 0.0499
Epoch 83/500
 - 0s - loss: 0.1597 - val_loss: 0.0495
Epoch 84/500
 - 0s - loss: 0.1581 - val_loss: 0.0491
Epoch 85/500
 - 0s - loss: 0.1564 - val_loss: 0.0489
Epoch 86/500
 - 0s - loss: 0.1546 - val_loss: 0.0483
Epoch 87/500
 - 0s - loss: 0.1528 - val_loss: 0.0477
Epoch 88/500
 - 0s - loss: 0.1509 - val_loss: 0.0471
Epoch 89/500
 - 0s - loss: 0.1489 - val_loss: 0.0469
Epoch 90/500
 - 0s - loss: 0.1469 - val_loss: 0.0470
Epoch 91/500
 - 0s - loss: 0.1447 - val_loss: 0.0471
Epoch 92/500
 - 0s - loss: 0.1425 - val_loss: 0.0472
Epoch 93/500
 - 0s - loss: 0.1402 - val_loss: 0.0470
Epoch 94/500
 - 0s - loss: 0.1378 - val_loss: 0.0468
Epoch 95/500
 - 0s - loss: 0.1353 - val_loss: 0.0466
Epoch 96/500
 - 0s - loss: 0.1328 - val_loss: 0.0460
Epoch 97/500
 - 0s - loss: 0.1303 - val_loss: 0.0456
Epoch 98/500
 - 0s - loss: 0.1277 - val_loss: 0.0452
Epoch 99/500
 - 0s - loss: 0.1250 - val_loss: 0.0446
Epoch 100/500
 - 0s - loss: 0.1222 - val_loss: 0.0442
Epoch 101/500
 - 0s - loss: 0.1193 - val_loss: 0.0440
Epoch 102/500
 - 0s - loss: 0.1164 - val_loss: 0.0437
Epoch 103/500
 - 0s - loss: 0.1134 - val_loss: 0.0435
Epoch 104/500
 - 0s - loss: 0.1103 - val_loss: 0.0435
Epoch 105/500
 - 0s - loss: 0.1071 - val_loss: 0.0438
Epoch 106/500
 - 0s - loss: 0.1039 - val_loss: 0.0443
Epoch 107/500
 - 0s - loss: 0.1007 - val_loss: 0.0448
Epoch 108/500
 - 0s - loss: 0.0976 - val_loss: 0.0453
Epoch 109/500
 - 0s - loss: 0.0945 - val_loss: 0.0459
Epoch 110/500
 - 0s - loss: 0.0915 - val_loss: 0.0467
Epoch 111/500
 - 0s - loss: 0.0886 - val_loss: 0.0477
Epoch 112/500
 - 0s - loss: 0.0856 - val_loss: 0.0486
Epoch 113/500
 - 0s - loss: 0.0827 - val_loss: 0.0500
Epoch 114/500
 - 0s - loss: 0.0804 - val_loss: 0.0517
Epoch 115/500
 - 0s - loss: 0.0783 - val_loss: 0.0534
Epoch 116/500
 - 0s - loss: 0.0761 - val_loss: 0.0539
Epoch 117/500
 - 0s - loss: 0.0747 - val_loss: 0.0551
Epoch 118/500
 - 0s - loss: 0.0732 - val_loss: 0.0562
Epoch 119/500
 - 0s - loss: 0.0717 - val_loss: 0.0570
Epoch 120/500
 - 0s - loss: 0.0705 - val_loss: 0.0583
Epoch 121/500
 - 0s - loss: 0.0693 - val_loss: 0.0590
Epoch 122/500
 - 0s - loss: 0.0679 - val_loss: 0.0589
Epoch 123/500
 - 0s - loss: 0.0670 - val_loss: 0.0597
Epoch 124/500
 - 0s - loss: 0.0660 - val_loss: 0.0602
Epoch 125/500
 - 0s - loss: 0.0649 - val_loss: 0.0599
Epoch 126/500
 - 0s - loss: 0.0640 - val_loss: 0.0601
Epoch 127/500
 - 0s - loss: 0.0630 - val_loss: 0.0599
Epoch 128/500
 - 0s - loss: 0.0621 - val_loss: 0.0597
Epoch 129/500
 - 0s - loss: 0.0614 - val_loss: 0.0594
Epoch 130/500
 - 0s - loss: 0.0606 - val_loss: 0.0593
Epoch 131/500
 - 0s - loss: 0.0598 - val_loss: 0.0592
Epoch 132/500
 - 0s - loss: 0.0590 - val_loss: 0.0587
Epoch 133/500
 - 0s - loss: 0.0583 - val_loss: 0.0588
Epoch 134/500
 - 0s - loss: 0.0576 - val_loss: 0.0586
Epoch 135/500
 - 0s - loss: 0.0568 - val_loss: 0.0582
Epoch 136/500
 - 0s - loss: 0.0561 - val_loss: 0.0579
Epoch 137/500
 - 0s - loss: 0.0554 - val_loss: 0.0573
Epoch 138/500
 - 0s - loss: 0.0548 - val_loss: 0.0567
Epoch 139/500
 - 0s - loss: 0.0542 - val_loss: 0.0563
Epoch 140/500
 - 0s - loss: 0.0536 - val_loss: 0.0562
Epoch 141/500
 - 0s - loss: 0.0529 - val_loss: 0.0552
Epoch 142/500
 - 0s - loss: 0.0524 - val_loss: 0.0547
Epoch 143/500
 - 0s - loss: 0.0518 - val_loss: 0.0544
Epoch 144/500
 - 0s - loss: 0.0512 - val_loss: 0.0539
Epoch 145/500
 - 0s - loss: 0.0507 - val_loss: 0.0535
Epoch 146/500
 - 0s - loss: 0.0501 - val_loss: 0.0527
Epoch 147/500
 - 0s - loss: 0.0496 - val_loss: 0.0520
Epoch 148/500
 - 0s - loss: 0.0491 - val_loss: 0.0516
Epoch 149/500
 - 0s - loss: 0.0485 - val_loss: 0.0511
Epoch 150/500
 - 0s - loss: 0.0480 - val_loss: 0.0505
Epoch 151/500
 - 0s - loss: 0.0476 - val_loss: 0.0502
Epoch 152/500
 - 0s - loss: 0.0471 - val_loss: 0.0496
Epoch 153/500
 - 0s - loss: 0.0466 - val_loss: 0.0488
Epoch 154/500
 - 0s - loss: 0.0461 - val_loss: 0.0483
Epoch 155/500
 - 0s - loss: 0.0457 - val_loss: 0.0474
Epoch 156/500
 - 0s - loss: 0.0453 - val_loss: 0.0467
Epoch 157/500
 - 0s - loss: 0.0449 - val_loss: 0.0461
Epoch 158/500
 - 0s - loss: 0.0445 - val_loss: 0.0451
Epoch 159/500
 - 0s - loss: 0.0441 - val_loss: 0.0445
Epoch 160/500
 - 0s - loss: 0.0437 - val_loss: 0.0438
Epoch 161/500
 - 0s - loss: 0.0433 - val_loss: 0.0429
Epoch 162/500
 - 0s - loss: 0.0430 - val_loss: 0.0423
Epoch 163/500
 - 0s - loss: 0.0426 - val_loss: 0.0415
Epoch 164/500
 - 0s - loss: 0.0422 - val_loss: 0.0411
Epoch 165/500
 - 0s - loss: 0.0419 - val_loss: 0.0401
Epoch 166/500
 - 0s - loss: 0.0416 - val_loss: 0.0394
Epoch 167/500
 - 0s - loss: 0.0413 - val_loss: 0.0388
Epoch 168/500
 - 0s - loss: 0.0410 - val_loss: 0.0381
Epoch 169/500
 - 0s - loss: 0.0407 - val_loss: 0.0377
Epoch 170/500
 - 0s - loss: 0.0404 - val_loss: 0.0374
Epoch 171/500
 - 0s - loss: 0.0401 - val_loss: 0.0365
Epoch 172/500
 - 0s - loss: 0.0399 - val_loss: 0.0361
Epoch 173/500
 - 0s - loss: 0.0396 - val_loss: 0.0359
Epoch 174/500
 - 0s - loss: 0.0394 - val_loss: 0.0352
Epoch 175/500
 - 0s - loss: 0.0392 - val_loss: 0.0348
Epoch 176/500
 - 0s - loss: 0.0389 - val_loss: 0.0344
Epoch 177/500
 - 0s - loss: 0.0387 - val_loss: 0.0336
Epoch 178/500
 - 0s - loss: 0.0385 - val_loss: 0.0331
Epoch 179/500
 - 0s - loss: 0.0383 - val_loss: 0.0324
Epoch 180/500
 - 0s - loss: 0.0381 - val_loss: 0.0322
Epoch 181/500
 - 0s - loss: 0.0379 - val_loss: 0.0309
Epoch 182/500
 - 0s - loss: 0.0377 - val_loss: 0.0312
Epoch 183/500
 - 0s - loss: 0.0375 - val_loss: 0.0302
Epoch 184/500
 - 0s - loss: 0.0373 - val_loss: 0.0306
Epoch 185/500
 - 0s - loss: 0.0371 - val_loss: 0.0298
Epoch 186/500
 - 0s - loss: 0.0370 - val_loss: 0.0297
Epoch 187/500
 - 0s - loss: 0.0369 - val_loss: 0.0295
Epoch 188/500
 - 0s - loss: 0.0367 - val_loss: 0.0291
Epoch 189/500
 - 0s - loss: 0.0366 - val_loss: 0.0285
Epoch 190/500
 - 0s - loss: 0.0364 - val_loss: 0.0282
Epoch 191/500
 - 0s - loss: 0.0364 - val_loss: 0.0280
Epoch 192/500
 - 0s - loss: 0.0362 - val_loss: 0.0279
Epoch 193/500
 - 0s - loss: 0.0361 - val_loss: 0.0267
Epoch 194/500
 - 0s - loss: 0.0360 - val_loss: 0.0271
Epoch 195/500
 - 0s - loss: 0.0359 - val_loss: 0.0263
Epoch 196/500
 - 0s - loss: 0.0357 - val_loss: 0.0260
Epoch 197/500
 - 0s - loss: 0.0356 - val_loss: 0.0259
Epoch 198/500
 - 0s - loss: 0.0355 - val_loss: 0.0255
Epoch 199/500
 - 0s - loss: 0.0354 - val_loss: 0.0252
Epoch 200/500
 - 0s - loss: 0.0353 - val_loss: 0.0247
Epoch 201/500
 - 0s - loss: 0.0352 - val_loss: 0.0244
Epoch 202/500
 - 0s - loss: 0.0351 - val_loss: 0.0242
Epoch 203/500
 - 0s - loss: 0.0350 - val_loss: 0.0237
Epoch 204/500
 - 0s - loss: 0.0349 - val_loss: 0.0235
Epoch 205/500
 - 0s - loss: 0.0348 - val_loss: 0.0233
Epoch 206/500
 - 0s - loss: 0.0346 - val_loss: 0.0227
Epoch 207/500
 - 0s - loss: 0.0345 - val_loss: 0.0223
Epoch 208/500
 - 0s - loss: 0.0344 - val_loss: 0.0221
Epoch 209/500
 - 0s - loss: 0.0343 - val_loss: 0.0219
Epoch 210/500
 - 0s - loss: 0.0343 - val_loss: 0.0217
Epoch 211/500
 - 0s - loss: 0.0342 - val_loss: 0.0214
Epoch 212/500
 - 0s - loss: 0.0341 - val_loss: 0.0213
Epoch 213/500
 - 0s - loss: 0.0340 - val_loss: 0.0209
Epoch 214/500
 - 0s - loss: 0.0339 - val_loss: 0.0207
Epoch 215/500
 - 0s - loss: 0.0338 - val_loss: 0.0207
Epoch 216/500
 - 0s - loss: 0.0337 - val_loss: 0.0204
Epoch 217/500
 - 0s - loss: 0.0337 - val_loss: 0.0202
Epoch 218/500
 - 0s - loss: 0.0336 - val_loss: 0.0201
Epoch 219/500
 - 0s - loss: 0.0335 - val_loss: 0.0199
Epoch 220/500
 - 0s - loss: 0.0334 - val_loss: 0.0196
Epoch 221/500
 - 0s - loss: 0.0334 - val_loss: 0.0195
Epoch 222/500
 - 0s - loss: 0.0333 - val_loss: 0.0194
Epoch 223/500
 - 0s - loss: 0.0332 - val_loss: 0.0193
Epoch 224/500
 - 0s - loss: 0.0332 - val_loss: 0.0194
Epoch 225/500
 - 0s - loss: 0.0331 - val_loss: 0.0192
Epoch 226/500
 - 0s - loss: 0.0330 - val_loss: 0.0191
Epoch 227/500
 - 0s - loss: 0.0329 - val_loss: 0.0190
Epoch 228/500
 - 0s - loss: 0.0329 - val_loss: 0.0189
Epoch 229/500
 - 0s - loss: 0.0328 - val_loss: 0.0189
Epoch 230/500
 - 0s - loss: 0.0327 - val_loss: 0.0187
Epoch 231/500
 - 0s - loss: 0.0327 - val_loss: 0.0188
Epoch 232/500
 - 0s - loss: 0.0326 - val_loss: 0.0185
Epoch 233/500
 - 0s - loss: 0.0326 - val_loss: 0.0187
Epoch 234/500
 - 0s - loss: 0.0325 - val_loss: 0.0183
Epoch 235/500
 - 0s - loss: 0.0325 - val_loss: 0.0184
Epoch 236/500
 - 0s - loss: 0.0325 - val_loss: 0.0183
Epoch 237/500
 - 0s - loss: 0.0324 - val_loss: 0.0182
Epoch 238/500
 - 0s - loss: 0.0324 - val_loss: 0.0180
Epoch 239/500
 - 0s - loss: 0.0323 - val_loss: 0.0184
Epoch 240/500
 - 0s - loss: 0.0323 - val_loss: 0.0178
Epoch 241/500
 - 0s - loss: 0.0322 - val_loss: 0.0181
Epoch 242/500
 - 0s - loss: 0.0321 - val_loss: 0.0177
Epoch 243/500
 - 0s - loss: 0.0321 - val_loss: 0.0178
Epoch 244/500
 - 0s - loss: 0.0321 - val_loss: 0.0178
Epoch 245/500
 - 0s - loss: 0.0320 - val_loss: 0.0178
Epoch 246/500
 - 0s - loss: 0.0320 - val_loss: 0.0174
Epoch 247/500
 - 0s - loss: 0.0319 - val_loss: 0.0179
Epoch 248/500
 - 0s - loss: 0.0319 - val_loss: 0.0173
Epoch 249/500
 - 0s - loss: 0.0318 - val_loss: 0.0174
Epoch 250/500
 - 0s - loss: 0.0318 - val_loss: 0.0172
Epoch 251/500
 - 0s - loss: 0.0317 - val_loss: 0.0174
Epoch 252/500
 - 0s - loss: 0.0317 - val_loss: 0.0172
Epoch 253/500
 - 0s - loss: 0.0316 - val_loss: 0.0170
Epoch 254/500
 - 0s - loss: 0.0316 - val_loss: 0.0170
Epoch 255/500
 - 0s - loss: 0.0316 - val_loss: 0.0171
Epoch 256/500
 - 0s - loss: 0.0314 - val_loss: 0.0168
Epoch 257/500
 - 0s - loss: 0.0314 - val_loss: 0.0168
Epoch 258/500
 - 0s - loss: 0.0314 - val_loss: 0.0168
Epoch 259/500
 - 0s - loss: 0.0313 - val_loss: 0.0166
Epoch 260/500
 - 0s - loss: 0.0313 - val_loss: 0.0166
Epoch 261/500
 - 0s - loss: 0.0312 - val_loss: 0.0166
Epoch 262/500
 - 0s - loss: 0.0312 - val_loss: 0.0165
Epoch 263/500
 - 0s - loss: 0.0312 - val_loss: 0.0166
Epoch 264/500
 - 0s - loss: 0.0311 - val_loss: 0.0164
Epoch 265/500
 - 0s - loss: 0.0310 - val_loss: 0.0162
Epoch 266/500
 - 0s - loss: 0.0310 - val_loss: 0.0167
Epoch 267/500
 - 0s - loss: 0.0310 - val_loss: 0.0162
Epoch 268/500
 - 0s - loss: 0.0310 - val_loss: 0.0163
Epoch 269/500
 - 0s - loss: 0.0309 - val_loss: 0.0159
Epoch 270/500
 - 0s - loss: 0.0309 - val_loss: 0.0161
Epoch 271/500
 - 0s - loss: 0.0309 - val_loss: 0.0161
Epoch 272/500
 - 0s - loss: 0.0308 - val_loss: 0.0160
Epoch 273/500
 - 0s - loss: 0.0307 - val_loss: 0.0156
Epoch 274/500
 - 0s - loss: 0.0307 - val_loss: 0.0164
Epoch 275/500
 - 0s - loss: 0.0307 - val_loss: 0.0159
Epoch 276/500
 - 0s - loss: 0.0306 - val_loss: 0.0157
Epoch 277/500
 - 0s - loss: 0.0306 - val_loss: 0.0161
Epoch 278/500
 - 0s - loss: 0.0306 - val_loss: 0.0160
Epoch 279/500
 - 0s - loss: 0.0305 - val_loss: 0.0159
Epoch 280/500
 - 0s - loss: 0.0304 - val_loss: 0.0156
Epoch 281/500
 - 0s - loss: 0.0304 - val_loss: 0.0160
Epoch 282/500
 - 0s - loss: 0.0304 - val_loss: 0.0158
Epoch 283/500
 - 0s - loss: 0.0303 - val_loss: 0.0157
Epoch 284/500
 - 0s - loss: 0.0303 - val_loss: 0.0158
Epoch 285/500
 - 0s - loss: 0.0303 - val_loss: 0.0158
Epoch 286/500
 - 0s - loss: 0.0303 - val_loss: 0.0156
Epoch 287/500
 - 0s - loss: 0.0302 - val_loss: 0.0157
Epoch 288/500
 - 0s - loss: 0.0302 - val_loss: 0.0156
Epoch 289/500
 - 0s - loss: 0.0301 - val_loss: 0.0155
Epoch 290/500
 - 0s - loss: 0.0302 - val_loss: 0.0156
Epoch 291/500
 - 0s - loss: 0.0300 - val_loss: 0.0154
Epoch 292/500
 - 0s - loss: 0.0301 - val_loss: 0.0157
Epoch 293/500
 - 0s - loss: 0.0300 - val_loss: 0.0154
Epoch 294/500
 - 0s - loss: 0.0300 - val_loss: 0.0155
Epoch 295/500
 - 0s - loss: 0.0299 - val_loss: 0.0153
Epoch 296/500
 - 0s - loss: 0.0299 - val_loss: 0.0154
Epoch 297/500
 - 0s - loss: 0.0298 - val_loss: 0.0152
Epoch 298/500
 - 0s - loss: 0.0299 - val_loss: 0.0153
Epoch 299/500
 - 0s - loss: 0.0298 - val_loss: 0.0152
Epoch 300/500
 - 0s - loss: 0.0298 - val_loss: 0.0153
Epoch 301/500
 - 0s - loss: 0.0297 - val_loss: 0.0153
Epoch 302/500
 - 0s - loss: 0.0297 - val_loss: 0.0152
Epoch 303/500
 - 0s - loss: 0.0297 - val_loss: 0.0151
Epoch 304/500
 - 0s - loss: 0.0296 - val_loss: 0.0151
Epoch 305/500
 - 0s - loss: 0.0296 - val_loss: 0.0152
Epoch 306/500
 - 0s - loss: 0.0296 - val_loss: 0.0151
Epoch 307/500
 - 0s - loss: 0.0295 - val_loss: 0.0151
Epoch 308/500
 - 0s - loss: 0.0295 - val_loss: 0.0150
Epoch 309/500
 - 0s - loss: 0.0295 - val_loss: 0.0151
Epoch 310/500
 - 0s - loss: 0.0294 - val_loss: 0.0150
Epoch 311/500
 - 0s - loss: 0.0294 - val_loss: 0.0151
Epoch 312/500
 - 0s - loss: 0.0293 - val_loss: 0.0149
Epoch 313/500
 - 0s - loss: 0.0293 - val_loss: 0.0151
Epoch 314/500
 - 0s - loss: 0.0293 - val_loss: 0.0150
Epoch 315/500
 - 0s - loss: 0.0293 - val_loss: 0.0150
Epoch 316/500
 - 0s - loss: 0.0292 - val_loss: 0.0150
Epoch 317/500
 - 0s - loss: 0.0292 - val_loss: 0.0150
Epoch 318/500
 - 0s - loss: 0.0291 - val_loss: 0.0150
Epoch 319/500
 - 0s - loss: 0.0292 - val_loss: 0.0149
Epoch 320/500
 - 0s - loss: 0.0291 - val_loss: 0.0149
Epoch 321/500
 - 0s - loss: 0.0290 - val_loss: 0.0150
Epoch 322/500
 - 0s - loss: 0.0291 - val_loss: 0.0149
Epoch 323/500
 - 0s - loss: 0.0290 - val_loss: 0.0148
Epoch 324/500
 - 0s - loss: 0.0290 - val_loss: 0.0149
Epoch 325/500
 - 0s - loss: 0.0290 - val_loss: 0.0148
Epoch 326/500
 - 0s - loss: 0.0289 - val_loss: 0.0148
Epoch 327/500
 - 0s - loss: 0.0289 - val_loss: 0.0148
Epoch 328/500
 - 0s - loss: 0.0289 - val_loss: 0.0149
Epoch 329/500
 - 0s - loss: 0.0288 - val_loss: 0.0148
Epoch 330/500
 - 0s - loss: 0.0288 - val_loss: 0.0148
Epoch 331/500
 - 0s - loss: 0.0288 - val_loss: 0.0148
Epoch 332/500
 - 0s - loss: 0.0288 - val_loss: 0.0150
Epoch 333/500
 - 0s - loss: 0.0287 - val_loss: 0.0148
Epoch 334/500
 - 0s - loss: 0.0287 - val_loss: 0.0148
Epoch 335/500
 - 0s - loss: 0.0287 - val_loss: 0.0148
Epoch 336/500
 - 0s - loss: 0.0286 - val_loss: 0.0149
Epoch 337/500
 - 0s - loss: 0.0286 - val_loss: 0.0147
Epoch 338/500
 - 0s - loss: 0.0286 - val_loss: 0.0148
Epoch 339/500
 - 0s - loss: 0.0286 - val_loss: 0.0147
Epoch 340/500
 - 0s - loss: 0.0286 - val_loss: 0.0147
Epoch 341/500
 - 0s - loss: 0.0285 - val_loss: 0.0147
Epoch 342/500
 - 0s - loss: 0.0285 - val_loss: 0.0146
Epoch 343/500
 - 0s - loss: 0.0285 - val_loss: 0.0146
Epoch 344/500
 - 0s - loss: 0.0285 - val_loss: 0.0145
Epoch 345/500
 - 0s - loss: 0.0285 - val_loss: 0.0146
Epoch 346/500
 - 0s - loss: 0.0284 - val_loss: 0.0146
Epoch 347/500
 - 0s - loss: 0.0284 - val_loss: 0.0146
Epoch 348/500
 - 0s - loss: 0.0284 - val_loss: 0.0145
Epoch 349/500
 - 0s - loss: 0.0284 - val_loss: 0.0146
Epoch 350/500
 - 0s - loss: 0.0283 - val_loss: 0.0146
Epoch 351/500
 - 0s - loss: 0.0283 - val_loss: 0.0145
Epoch 352/500
 - 0s - loss: 0.0283 - val_loss: 0.0144
Epoch 353/500
 - 0s - loss: 0.0283 - val_loss: 0.0144
Epoch 354/500
 - 0s - loss: 0.0283 - val_loss: 0.0145
Epoch 355/500
 - 0s - loss: 0.0283 - val_loss: 0.0145
Epoch 356/500
 - 0s - loss: 0.0282 - val_loss: 0.0144
Epoch 357/500
 - 0s - loss: 0.0282 - val_loss: 0.0144
Epoch 358/500
 - 0s - loss: 0.0282 - val_loss: 0.0143
Epoch 359/500
 - 0s - loss: 0.0282 - val_loss: 0.0144
Epoch 360/500
 - 0s - loss: 0.0282 - val_loss: 0.0143
Epoch 361/500
 - 0s - loss: 0.0282 - val_loss: 0.0144
Epoch 362/500
 - 0s - loss: 0.0281 - val_loss: 0.0143
Epoch 363/500
 - 0s - loss: 0.0281 - val_loss: 0.0143
Epoch 364/500
 - 0s - loss: 0.0281 - val_loss: 0.0143
Epoch 365/500
 - 0s - loss: 0.0281 - val_loss: 0.0142
Epoch 366/500
 - 0s - loss: 0.0281 - val_loss: 0.0142
Epoch 367/500
 - 0s - loss: 0.0281 - val_loss: 0.0142
Epoch 368/500
 - 0s - loss: 0.0280 - val_loss: 0.0141
Epoch 369/500
 - 0s - loss: 0.0281 - val_loss: 0.0144
Epoch 370/500
 - 0s - loss: 0.0280 - val_loss: 0.0141
Epoch 371/500
 - 0s - loss: 0.0280 - val_loss: 0.0141
Epoch 372/500
 - 0s - loss: 0.0280 - val_loss: 0.0141
Epoch 373/500
 - 0s - loss: 0.0279 - val_loss: 0.0140
Epoch 374/500
 - 0s - loss: 0.0279 - val_loss: 0.0141
Epoch 375/500
 - 0s - loss: 0.0279 - val_loss: 0.0141
Epoch 376/500
 - 0s - loss: 0.0279 - val_loss: 0.0140
Epoch 377/500
 - 0s - loss: 0.0279 - val_loss: 0.0142
Epoch 378/500
 - 0s - loss: 0.0279 - val_loss: 0.0143
Epoch 379/500
 - 0s - loss: 0.0278 - val_loss: 0.0139
Epoch 380/500
 - 0s - loss: 0.0279 - val_loss: 0.0143
Epoch 381/500
 - 0s - loss: 0.0278 - val_loss: 0.0141
Epoch 382/500
 - 0s - loss: 0.0278 - val_loss: 0.0139
Epoch 383/500
 - 0s - loss: 0.0278 - val_loss: 0.0142
Epoch 384/500
 - 0s - loss: 0.0278 - val_loss: 0.0141
Epoch 385/500
 - 0s - loss: 0.0277 - val_loss: 0.0140
Epoch 386/500
 - 0s - loss: 0.0277 - val_loss: 0.0139
Epoch 387/500
 - 0s - loss: 0.0277 - val_loss: 0.0143
Epoch 388/500
 - 0s - loss: 0.0277 - val_loss: 0.0142
Epoch 389/500
 - 0s - loss: 0.0277 - val_loss: 0.0139
Epoch 390/500
 - 0s - loss: 0.0277 - val_loss: 0.0139
Epoch 391/500
 - 0s - loss: 0.0277 - val_loss: 0.0141
Epoch 392/500
 - 0s - loss: 0.0277 - val_loss: 0.0142
Epoch 393/500
 - 0s - loss: 0.0276 - val_loss: 0.0139
Epoch 394/500
 - 0s - loss: 0.0276 - val_loss: 0.0140
Epoch 395/500
 - 0s - loss: 0.0276 - val_loss: 0.0142
Epoch 396/500
 - 0s - loss: 0.0276 - val_loss: 0.0143
Epoch 397/500
 - 0s - loss: 0.0276 - val_loss: 0.0140
Epoch 398/500
 - 0s - loss: 0.0276 - val_loss: 0.0140
Epoch 399/500
 - 0s - loss: 0.0276 - val_loss: 0.0139
Epoch 400/500
 - 0s - loss: 0.0275 - val_loss: 0.0140
Epoch 401/500
 - 0s - loss: 0.0275 - val_loss: 0.0140
Epoch 402/500
 - 0s - loss: 0.0275 - val_loss: 0.0139
Epoch 403/500
 - 0s - loss: 0.0275 - val_loss: 0.0140
Epoch 404/500
 - 0s - loss: 0.0275 - val_loss: 0.0138
Epoch 405/500
 - 0s - loss: 0.0275 - val_loss: 0.0141
Epoch 406/500
 - 0s - loss: 0.0275 - val_loss: 0.0141
Epoch 407/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 408/500
 - 0s - loss: 0.0274 - val_loss: 0.0137
Epoch 409/500
 - 0s - loss: 0.0275 - val_loss: 0.0144
Epoch 410/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 411/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 412/500
 - 0s - loss: 0.0274 - val_loss: 0.0138
Epoch 413/500
 - 0s - loss: 0.0274 - val_loss: 0.0140
Epoch 414/500
 - 0s - loss: 0.0274 - val_loss: 0.0138
Epoch 415/500
 - 0s - loss: 0.0274 - val_loss: 0.0138
Epoch 416/500
 - 0s - loss: 0.0274 - val_loss: 0.0138
Epoch 417/500
 - 0s - loss: 0.0274 - val_loss: 0.0140
Epoch 418/500
 - 0s - loss: 0.0273 - val_loss: 0.0138
Epoch 419/500
 - 0s - loss: 0.0273 - val_loss: 0.0138
Epoch 420/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 421/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 422/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 423/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 424/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 425/500
 - 0s - loss: 0.0273 - val_loss: 0.0138
Epoch 426/500
 - 0s - loss: 0.0273 - val_loss: 0.0137
Epoch 427/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 428/500
 - 0s - loss: 0.0273 - val_loss: 0.0139
Epoch 429/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 430/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 431/500
 - 0s - loss: 0.0272 - val_loss: 0.0136
Epoch 432/500
 - 0s - loss: 0.0272 - val_loss: 0.0136
Epoch 433/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 434/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 435/500
 - 0s - loss: 0.0272 - val_loss: 0.0135
Epoch 436/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 437/500
 - 0s - loss: 0.0272 - val_loss: 0.0136
Epoch 438/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 439/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 440/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 441/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 442/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 443/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 444/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 445/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 446/500
 - 0s - loss: 0.0271 - val_loss: 0.0135
Epoch 447/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 448/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 449/500
 - 0s - loss: 0.0271 - val_loss: 0.0135
Epoch 450/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 451/500
 - 0s - loss: 0.0271 - val_loss: 0.0135
Epoch 452/500
 - 0s - loss: 0.0271 - val_loss: 0.0134
Epoch 453/500
 - 0s - loss: 0.0271 - val_loss: 0.0134
Epoch 454/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 455/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 456/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 457/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 458/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 459/500
 - 0s - loss: 0.0270 - val_loss: 0.0135
Epoch 460/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 461/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 462/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 463/500
 - 0s - loss: 0.0270 - val_loss: 0.0133
Epoch 464/500
 - 0s - loss: 0.0270 - val_loss: 0.0135
Epoch 465/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 466/500
 - 0s - loss: 0.0269 - val_loss: 0.0133
Epoch 467/500
 - 0s - loss: 0.0270 - val_loss: 0.0132
Epoch 468/500
 - 0s - loss: 0.0270 - val_loss: 0.0134
Epoch 469/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 470/500
 - 0s - loss: 0.0269 - val_loss: 0.0133
Epoch 471/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 472/500
 - 0s - loss: 0.0269 - val_loss: 0.0133
Epoch 473/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 474/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 475/500
 - 0s - loss: 0.0269 - val_loss: 0.0132
Epoch 476/500
 - 0s - loss: 0.0269 - val_loss: 0.0131
Epoch 477/500
 - 0s - loss: 0.0269 - val_loss: 0.0130
Epoch 478/500
 - 0s - loss: 0.0269 - val_loss: 0.0131
Epoch 479/500
 - 0s - loss: 0.0269 - val_loss: 0.0131
Epoch 480/500
 - 0s - loss: 0.0269 - val_loss: 0.0131
Epoch 481/500
 - 0s - loss: 0.0268 - val_loss: 0.0132
Epoch 482/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 483/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 484/500
 - 0s - loss: 0.0268 - val_loss: 0.0130
Epoch 485/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 486/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 487/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 488/500
 - 0s - loss: 0.0268 - val_loss: 0.0131
Epoch 489/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 490/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 491/500
 - 0s - loss: 0.0268 - val_loss: 0.0130
Epoch 492/500
 - 0s - loss: 0.0268 - val_loss: 0.0130
Epoch 493/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 494/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 495/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 496/500
 - 0s - loss: 0.0268 - val_loss: 0.0129
Epoch 497/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 498/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 499/500
 - 0s - loss: 0.0267 - val_loss: 0.0128
Epoch 500/500
 - 0s - loss: 0.0267 - val_loss: 0.0130
In [68]:
pyplot.plot(history['loss'], label='train')
pyplot.plot(history['val_loss'], label='validation')
pyplot.legend()
pyplot.show()
In [69]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(validation_X,validation_y,model,scaler)
print('LSTM Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Model on Validation Data RMSE: 9.479
In [70]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_series_to_compare(inv_y,inv_yhat,"Actual Price","Predicted Price", "Actual Price Versus LSTM Predicted Price")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

7. Bench Mark Model

In this section we will check our bench mark model. As is proposed in my proposal my bench mark model is a simple linear regressor model.

Load the preprocessed data

In [34]:
from pandas import read_csv
from pandas import datetime
from pandas import DataFrame
from pandas import concat
from matplotlib import pyplot
from sklearn.metrics import mean_squared_error
from math import sqrt


# Create lagged dataset
values = pd.DataFrame(df_weekly["Settle"].values)
df_benchmark  = concat([values.shift(1), values], axis=1)
df_benchmark.columns = ['t', 't+1']
display(df_benchmark.head(5))
t t+1
0 NaN 235.50
1 235.50 228.25
2 228.25 235.50
3 235.50 241.00
4 241.00 253.50
In [35]:
# split into train , validation and test sets
X = df_benchmark.values
train, validation, test = X[1:validation_start], X[validation_start:testing_start],X[testing_start:]
train_bench_X, train_bench_y = train[:,0], train[:,1]
validation_bench_X, validation_bench_y = validation[:,0], validation[:,1]
test_bench_X, test_bench_y = test[:,0], test[:,1]
In [36]:
%load_ext autoreload
%autoreload 2
import models
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [37]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
predictions,rmse=models.make_benchmark_model_prediction(validation_bench_X,validation_bench_y)
print('Benchmark Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Benchmark Model on Validation Data RMSE: 8.750
In [38]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_series_to_compare(validation_bench_y,predictions,"Actual Price","Predicted Price", "Actual Price Versus Benchmark Model Predicted Price")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

8. Test model on unseen data

Test LSTM model on unseen data

In [71]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(test_X,test_y,model,scaler)
print('LSTM Moddel on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Moddel on Test Data RMSE: 12.079

Test Benchmark model on unseen data

In [72]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
predictions,rmse=models.make_benchmark_model_prediction(test_bench_X,test_bench_y)
print('Benchmark Model on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Benchmark Model on Test Data RMSE: 8.293

9. Tune basic LSTM Model

In [41]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_memmory_cells(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=1.000000, loss=0.012129
>2/5 param=1.000000, loss=0.014510
>3/5 param=1.000000, loss=0.011251
>4/5 param=1.000000, loss=0.013165
>5/5 param=1.000000, loss=0.012103
>1/5 param=5.000000, loss=0.011613
>2/5 param=5.000000, loss=0.012066
>3/5 param=5.000000, loss=0.011987
>4/5 param=5.000000, loss=0.012024
>5/5 param=5.000000, loss=0.012577
>1/5 param=10.000000, loss=0.012330
>2/5 param=10.000000, loss=0.013115
>3/5 param=10.000000, loss=0.013052
>4/5 param=10.000000, loss=0.011792
>5/5 param=10.000000, loss=0.013219
>1/5 param=25.000000, loss=0.011451
>2/5 param=25.000000, loss=0.013046
>3/5 param=25.000000, loss=0.011217
>4/5 param=25.000000, loss=0.011381
>5/5 param=25.000000, loss=0.011058
>1/5 param=50.000000, loss=0.012644
>2/5 param=50.000000, loss=0.012646
>3/5 param=50.000000, loss=0.011140
>4/5 param=50.000000, loss=0.013345
>5/5 param=50.000000, loss=0.012515
>1/5 param=100.000000, loss=0.011604
>2/5 param=100.000000, loss=0.015141
>3/5 param=100.000000, loss=0.012493
>4/5 param=100.000000, loss=0.012222
>5/5 param=100.000000, loss=0.013316
>1/5 param=200.000000, loss=0.011432
>2/5 param=200.000000, loss=0.012943
>3/5 param=200.000000, loss=0.010766
>4/5 param=200.000000, loss=0.014985
>5/5 param=200.000000, loss=0.011893
              1         5        10        25        50       100       200
count  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000
mean   0.012632  0.012053  0.012702  0.011631  0.012458  0.012955  0.012404
std    0.001250  0.000344  0.000618  0.000806  0.000805  0.001368  0.001646
min    0.011251  0.011613  0.011792  0.011058  0.011140  0.011604  0.010766
25%    0.012103  0.011987  0.012330  0.011217  0.012515  0.012222  0.011432
50%    0.012129  0.012024  0.013052  0.011381  0.012644  0.012493  0.011893
75%    0.013165  0.012066  0.013115  0.011451  0.012646  0.013316  0.012943
max    0.014510  0.012577  0.013219  0.013046  0.013345  0.015141  0.014985
In [42]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_batch_size(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=2.000000, loss=0.017062
>2/5 param=2.000000, loss=0.017288
>3/5 param=2.000000, loss=0.019628
>4/5 param=2.000000, loss=0.017816
>5/5 param=2.000000, loss=0.019510
>1/5 param=4.000000, loss=0.012946
>2/5 param=4.000000, loss=0.013468
>3/5 param=4.000000, loss=0.012065
>4/5 param=4.000000, loss=0.012588
>5/5 param=4.000000, loss=0.013342
>1/5 param=8.000000, loss=0.014913
>2/5 param=8.000000, loss=0.016104
>3/5 param=8.000000, loss=0.015724
>4/5 param=8.000000, loss=0.015701
>5/5 param=8.000000, loss=0.014112
>1/5 param=32.000000, loss=0.011369
>2/5 param=32.000000, loss=0.011775
>3/5 param=32.000000, loss=0.012824
>4/5 param=32.000000, loss=0.012704
>5/5 param=32.000000, loss=0.011133
>1/5 param=64.000000, loss=0.011609
>2/5 param=64.000000, loss=0.011532
>3/5 param=64.000000, loss=0.013435
>4/5 param=64.000000, loss=0.011951
>5/5 param=64.000000, loss=0.012349
>1/5 param=128.000000, loss=0.011928
>2/5 param=128.000000, loss=0.012988
>3/5 param=128.000000, loss=0.011940
>4/5 param=128.000000, loss=0.011974
>5/5 param=128.000000, loss=0.011488
>1/5 param=256.000000, loss=0.011780
>2/5 param=256.000000, loss=0.013215
>3/5 param=256.000000, loss=0.012355
>4/5 param=256.000000, loss=0.011390
>5/5 param=256.000000, loss=0.011595
              2         4         8        32        64       128       256
count  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000
mean   0.018261  0.012882  0.015311  0.011961  0.012175  0.012064  0.012067
std    0.001226  0.000573  0.000798  0.000769  0.000775  0.000554  0.000736
min    0.017062  0.012065  0.014112  0.011133  0.011532  0.011488  0.011390
25%    0.017288  0.012588  0.014913  0.011369  0.011609  0.011928  0.011595
50%    0.017816  0.012946  0.015701  0.011775  0.011951  0.011940  0.011780
75%    0.019510  0.013342  0.015724  0.012704  0.012349  0.011974  0.012355
max    0.019628  0.013468  0.016104  0.012824  0.013435  0.012988  0.013215
In [85]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_learning_rate(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=0.100000, loss=0.011195
>2/5 param=0.100000, loss=0.011809
>3/5 param=0.100000, loss=0.018709
>4/5 param=0.100000, loss=0.032624
>5/5 param=0.100000, loss=0.015232
>1/5 param=0.001000, loss=0.011950
>2/5 param=0.001000, loss=0.012022
>3/5 param=0.001000, loss=0.012852
>4/5 param=0.001000, loss=0.012499
>5/5 param=0.001000, loss=0.011699
>1/5 param=0.000100, loss=0.033569
>2/5 param=0.000100, loss=0.027969
>3/5 param=0.000100, loss=0.051960
>4/5 param=0.000100, loss=0.043928
>5/5 param=0.000100, loss=0.035140
            0.1     0.001    0.0001
count  5.000000  5.000000  5.000000
mean   0.017914  0.012204  0.038513
std    0.008756  0.000464  0.009449
min    0.011195  0.011699  0.027969
25%    0.011809  0.011950  0.033569
50%    0.015232  0.012022  0.035140
75%    0.018709  0.012499  0.043928
max    0.032624  0.012852  0.051960
In [86]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_weight_regularization(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=1.000000, loss=0.017913
>2/5 param=1.000000, loss=0.017820
>3/5 param=1.000000, loss=0.017644
>4/5 param=1.000000, loss=0.018630
>5/5 param=1.000000, loss=0.018884
>1/5 param=2.000000, loss=0.033898
>2/5 param=2.000000, loss=0.035481
>3/5 param=2.000000, loss=0.036565
>4/5 param=2.000000, loss=0.036193
>5/5 param=2.000000, loss=0.035109
>1/5 param=3.000000, loss=0.012555
>2/5 param=3.000000, loss=0.011829
>3/5 param=3.000000, loss=0.012490
>4/5 param=3.000000, loss=0.011938
>5/5 param=3.000000, loss=0.011889
>1/5 param=4.000000, loss=0.037587
>2/5 param=4.000000, loss=0.039391
>3/5 param=4.000000, loss=0.038415
>4/5 param=4.000000, loss=0.039397
>5/5 param=4.000000, loss=0.038773
              1         2         3         4
count  5.000000  5.000000  5.000000  5.000000
mean   0.018178  0.035449  0.012140  0.038713
std    0.000544  0.001040  0.000352  0.000756
min    0.017644  0.033898  0.011829  0.037587
25%    0.017820  0.035109  0.011889  0.038415
50%    0.017913  0.035481  0.011938  0.038773
75%    0.018630  0.036193  0.012490  0.039391
max    0.018884  0.036565  0.012555  0.039397

10. Test Improved LSTM Model

In [87]:
%load_ext autoreload
%autoreload 2
import models
model,history=models.improved_lstm_model(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Train on 550 samples, validate on 53 samples
Epoch 1/500
 - 21s - loss: 0.7108 - val_loss: 0.5045
Epoch 2/500
 - 0s - loss: 0.6787 - val_loss: 0.4682
Epoch 3/500
 - 0s - loss: 0.6448 - val_loss: 0.4312
Epoch 4/500
 - 0s - loss: 0.6109 - val_loss: 0.3942
Epoch 5/500
 - 0s - loss: 0.5775 - val_loss: 0.3571
Epoch 6/500
 - 0s - loss: 0.5447 - val_loss: 0.3210
Epoch 7/500
 - 0s - loss: 0.5117 - val_loss: 0.2875
Epoch 8/500
 - 0s - loss: 0.4798 - val_loss: 0.2657
Epoch 9/500
 - 0s - loss: 0.4515 - val_loss: 0.2674
Epoch 10/500
 - 0s - loss: 0.4297 - val_loss: 0.2824
Epoch 11/500
 - 0s - loss: 0.4149 - val_loss: 0.2977
Epoch 12/500
 - 0s - loss: 0.4052 - val_loss: 0.3093
Epoch 13/500
 - 0s - loss: 0.3984 - val_loss: 0.3167
Epoch 14/500
 - 0s - loss: 0.3934 - val_loss: 0.3195
Epoch 15/500
 - 0s - loss: 0.3892 - val_loss: 0.3193
Epoch 16/500
 - 0s - loss: 0.3854 - val_loss: 0.3171
Epoch 17/500
 - 0s - loss: 0.3819 - val_loss: 0.3138
Epoch 18/500
 - 0s - loss: 0.3786 - val_loss: 0.3097
Epoch 19/500
 - 0s - loss: 0.3753 - val_loss: 0.3053
Epoch 20/500
 - 0s - loss: 0.3721 - val_loss: 0.3009
Epoch 21/500
 - 0s - loss: 0.3689 - val_loss: 0.2964
Epoch 22/500
 - 0s - loss: 0.3657 - val_loss: 0.2918
Epoch 23/500
 - 0s - loss: 0.3626 - val_loss: 0.2874
Epoch 24/500
 - 0s - loss: 0.3595 - val_loss: 0.2833
Epoch 25/500
 - 0s - loss: 0.3563 - val_loss: 0.2793
Epoch 26/500
 - 0s - loss: 0.3531 - val_loss: 0.2754
Epoch 27/500
 - 0s - loss: 0.3500 - val_loss: 0.2717
Epoch 28/500
 - 0s - loss: 0.3468 - val_loss: 0.2682
Epoch 29/500
 - 0s - loss: 0.3436 - val_loss: 0.2647
Epoch 30/500
 - 0s - loss: 0.3404 - val_loss: 0.2614
Epoch 31/500
 - 0s - loss: 0.3372 - val_loss: 0.2581
Epoch 32/500
 - 0s - loss: 0.3340 - val_loss: 0.2549
Epoch 33/500
 - 0s - loss: 0.3308 - val_loss: 0.2518
Epoch 34/500
 - 0s - loss: 0.3276 - val_loss: 0.2488
Epoch 35/500
 - 0s - loss: 0.3243 - val_loss: 0.2458
Epoch 36/500
 - 0s - loss: 0.3211 - val_loss: 0.2430
Epoch 37/500
 - 0s - loss: 0.3178 - val_loss: 0.2401
Epoch 38/500
 - 0s - loss: 0.3145 - val_loss: 0.2371
Epoch 39/500
 - 0s - loss: 0.3113 - val_loss: 0.2342
Epoch 40/500
 - 0s - loss: 0.3080 - val_loss: 0.2312
Epoch 41/500
 - 0s - loss: 0.3048 - val_loss: 0.2281
Epoch 42/500
 - 0s - loss: 0.3015 - val_loss: 0.2252
Epoch 43/500
 - 0s - loss: 0.2982 - val_loss: 0.2223
Epoch 44/500
 - 0s - loss: 0.2949 - val_loss: 0.2195
Epoch 45/500
 - 0s - loss: 0.2916 - val_loss: 0.2167
Epoch 46/500
 - 0s - loss: 0.2883 - val_loss: 0.2139
Epoch 47/500
 - 0s - loss: 0.2850 - val_loss: 0.2110
Epoch 48/500
 - 0s - loss: 0.2816 - val_loss: 0.2081
Epoch 49/500
 - 0s - loss: 0.2783 - val_loss: 0.2052
Epoch 50/500
 - 0s - loss: 0.2749 - val_loss: 0.2022
Epoch 51/500
 - 0s - loss: 0.2716 - val_loss: 0.1994
Epoch 52/500
 - 0s - loss: 0.2682 - val_loss: 0.1968
Epoch 53/500
 - 0s - loss: 0.2647 - val_loss: 0.1942
Epoch 54/500
 - 0s - loss: 0.2612 - val_loss: 0.1915
Epoch 55/500
 - 0s - loss: 0.2577 - val_loss: 0.1886
Epoch 56/500
 - 0s - loss: 0.2542 - val_loss: 0.1859
Epoch 57/500
 - 0s - loss: 0.2507 - val_loss: 0.1832
Epoch 58/500
 - 0s - loss: 0.2471 - val_loss: 0.1805
Epoch 59/500
 - 0s - loss: 0.2436 - val_loss: 0.1776
Epoch 60/500
 - 0s - loss: 0.2400 - val_loss: 0.1746
Epoch 61/500
 - 0s - loss: 0.2364 - val_loss: 0.1714
Epoch 62/500
 - 0s - loss: 0.2328 - val_loss: 0.1683
Epoch 63/500
 - 0s - loss: 0.2292 - val_loss: 0.1652
Epoch 64/500
 - 0s - loss: 0.2256 - val_loss: 0.1621
Epoch 65/500
 - 0s - loss: 0.2219 - val_loss: 0.1588
Epoch 66/500
 - 0s - loss: 0.2182 - val_loss: 0.1555
Epoch 67/500
 - 0s - loss: 0.2145 - val_loss: 0.1522
Epoch 68/500
 - 0s - loss: 0.2107 - val_loss: 0.1496
Epoch 69/500
 - 0s - loss: 0.2068 - val_loss: 0.1475
Epoch 70/500
 - 0s - loss: 0.2028 - val_loss: 0.1457
Epoch 71/500
 - 0s - loss: 0.1987 - val_loss: 0.1435
Epoch 72/500
 - 0s - loss: 0.1948 - val_loss: 0.1407
Epoch 73/500
 - 0s - loss: 0.1908 - val_loss: 0.1377
Epoch 74/500
 - 0s - loss: 0.1870 - val_loss: 0.1345
Epoch 75/500
 - 0s - loss: 0.1832 - val_loss: 0.1318
Epoch 76/500
 - 0s - loss: 0.1792 - val_loss: 0.1297
Epoch 77/500
 - 0s - loss: 0.1752 - val_loss: 0.1277
Epoch 78/500
 - 0s - loss: 0.1713 - val_loss: 0.1256
Epoch 79/500
 - 0s - loss: 0.1675 - val_loss: 0.1236
Epoch 80/500
 - 0s - loss: 0.1639 - val_loss: 0.1215
Epoch 81/500
 - 0s - loss: 0.1603 - val_loss: 0.1200
Epoch 82/500
 - 0s - loss: 0.1568 - val_loss: 0.1190
Epoch 83/500
 - 0s - loss: 0.1533 - val_loss: 0.1180
Epoch 84/500
 - 0s - loss: 0.1500 - val_loss: 0.1170
Epoch 85/500
 - 0s - loss: 0.1468 - val_loss: 0.1161
Epoch 86/500
 - 0s - loss: 0.1439 - val_loss: 0.1155
Epoch 87/500
 - 0s - loss: 0.1409 - val_loss: 0.1146
Epoch 88/500
 - 0s - loss: 0.1380 - val_loss: 0.1136
Epoch 89/500
 - 0s - loss: 0.1353 - val_loss: 0.1127
Epoch 90/500
 - 0s - loss: 0.1329 - val_loss: 0.1123
Epoch 91/500
 - 0s - loss: 0.1305 - val_loss: 0.1117
Epoch 92/500
 - 0s - loss: 0.1281 - val_loss: 0.1106
Epoch 93/500
 - 0s - loss: 0.1260 - val_loss: 0.1097
Epoch 94/500
 - 0s - loss: 0.1240 - val_loss: 0.1091
Epoch 95/500
 - 0s - loss: 0.1221 - val_loss: 0.1080
Epoch 96/500
 - 0s - loss: 0.1203 - val_loss: 0.1069
Epoch 97/500
 - 0s - loss: 0.1186 - val_loss: 0.1055
Epoch 98/500
 - 0s - loss: 0.1170 - val_loss: 0.1043
Epoch 99/500
 - 0s - loss: 0.1154 - val_loss: 0.1033
Epoch 100/500
 - 0s - loss: 0.1138 - val_loss: 0.1020
Epoch 101/500
 - 0s - loss: 0.1123 - val_loss: 0.1006
Epoch 102/500
 - 0s - loss: 0.1109 - val_loss: 0.0990
Epoch 103/500
 - 0s - loss: 0.1095 - val_loss: 0.0976
Epoch 104/500
 - 0s - loss: 0.1081 - val_loss: 0.0964
Epoch 105/500
 - 0s - loss: 0.1068 - val_loss: 0.0954
Epoch 106/500
 - 0s - loss: 0.1054 - val_loss: 0.0941
Epoch 107/500
 - 0s - loss: 0.1041 - val_loss: 0.0925
Epoch 108/500
 - 0s - loss: 0.1029 - val_loss: 0.0910
Epoch 109/500
 - 0s - loss: 0.1016 - val_loss: 0.0897
Epoch 110/500
 - 0s - loss: 0.1004 - val_loss: 0.0885
Epoch 111/500
 - 0s - loss: 0.0992 - val_loss: 0.0873
Epoch 112/500
 - 0s - loss: 0.0981 - val_loss: 0.0858
Epoch 113/500
 - 0s - loss: 0.0970 - val_loss: 0.0847
Epoch 114/500
 - 0s - loss: 0.0959 - val_loss: 0.0837
Epoch 115/500
 - 0s - loss: 0.0948 - val_loss: 0.0823
Epoch 116/500
 - 0s - loss: 0.0937 - val_loss: 0.0811
Epoch 117/500
 - 0s - loss: 0.0927 - val_loss: 0.0802
Epoch 118/500
 - 0s - loss: 0.0917 - val_loss: 0.0791
Epoch 119/500
 - 0s - loss: 0.0907 - val_loss: 0.0777
Epoch 120/500
 - 0s - loss: 0.0897 - val_loss: 0.0765
Epoch 121/500
 - 0s - loss: 0.0888 - val_loss: 0.0756
Epoch 122/500
 - 0s - loss: 0.0878 - val_loss: 0.0745
Epoch 123/500
 - 0s - loss: 0.0869 - val_loss: 0.0734
Epoch 124/500
 - 0s - loss: 0.0860 - val_loss: 0.0725
Epoch 125/500
 - 0s - loss: 0.0851 - val_loss: 0.0715
Epoch 126/500
 - 0s - loss: 0.0842 - val_loss: 0.0704
Epoch 127/500
 - 0s - loss: 0.0833 - val_loss: 0.0696
Epoch 128/500
 - 0s - loss: 0.0825 - val_loss: 0.0687
Epoch 129/500
 - 0s - loss: 0.0816 - val_loss: 0.0676
Epoch 130/500
 - 0s - loss: 0.0808 - val_loss: 0.0666
Epoch 131/500
 - 0s - loss: 0.0800 - val_loss: 0.0658
Epoch 132/500
 - 0s - loss: 0.0792 - val_loss: 0.0650
Epoch 133/500
 - 0s - loss: 0.0784 - val_loss: 0.0641
Epoch 134/500
 - 0s - loss: 0.0776 - val_loss: 0.0631
Epoch 135/500
 - 0s - loss: 0.0768 - val_loss: 0.0624
Epoch 136/500
 - 0s - loss: 0.0761 - val_loss: 0.0616
Epoch 137/500
 - 0s - loss: 0.0753 - val_loss: 0.0608
Epoch 138/500
 - 0s - loss: 0.0746 - val_loss: 0.0600
Epoch 139/500
 - 0s - loss: 0.0739 - val_loss: 0.0592
Epoch 140/500
 - 0s - loss: 0.0732 - val_loss: 0.0585
Epoch 141/500
 - 0s - loss: 0.0725 - val_loss: 0.0576
Epoch 142/500
 - 0s - loss: 0.0718 - val_loss: 0.0569
Epoch 143/500
 - 0s - loss: 0.0711 - val_loss: 0.0562
Epoch 144/500
 - 0s - loss: 0.0704 - val_loss: 0.0554
Epoch 145/500
 - 0s - loss: 0.0697 - val_loss: 0.0547
Epoch 146/500
 - 0s - loss: 0.0691 - val_loss: 0.0541
Epoch 147/500
 - 0s - loss: 0.0684 - val_loss: 0.0533
Epoch 148/500
 - 0s - loss: 0.0678 - val_loss: 0.0526
Epoch 149/500
 - 0s - loss: 0.0672 - val_loss: 0.0518
Epoch 150/500
 - 0s - loss: 0.0665 - val_loss: 0.0512
Epoch 151/500
 - 0s - loss: 0.0659 - val_loss: 0.0505
Epoch 152/500
 - 0s - loss: 0.0653 - val_loss: 0.0499
Epoch 153/500
 - 0s - loss: 0.0647 - val_loss: 0.0492
Epoch 154/500
 - 0s - loss: 0.0641 - val_loss: 0.0486
Epoch 155/500
 - 0s - loss: 0.0635 - val_loss: 0.0480
Epoch 156/500
 - 0s - loss: 0.0630 - val_loss: 0.0474
Epoch 157/500
 - 0s - loss: 0.0624 - val_loss: 0.0468
Epoch 158/500
 - 0s - loss: 0.0618 - val_loss: 0.0462
Epoch 159/500
 - 0s - loss: 0.0613 - val_loss: 0.0456
Epoch 160/500
 - 0s - loss: 0.0607 - val_loss: 0.0450
Epoch 161/500
 - 0s - loss: 0.0602 - val_loss: 0.0445
Epoch 162/500
 - 0s - loss: 0.0597 - val_loss: 0.0440
Epoch 163/500
 - 0s - loss: 0.0591 - val_loss: 0.0434
Epoch 164/500
 - 0s - loss: 0.0586 - val_loss: 0.0428
Epoch 165/500
 - 0s - loss: 0.0581 - val_loss: 0.0424
Epoch 166/500
 - 0s - loss: 0.0576 - val_loss: 0.0419
Epoch 167/500
 - 0s - loss: 0.0571 - val_loss: 0.0413
Epoch 168/500
 - 0s - loss: 0.0566 - val_loss: 0.0408
Epoch 169/500
 - 0s - loss: 0.0561 - val_loss: 0.0404
Epoch 170/500
 - 0s - loss: 0.0556 - val_loss: 0.0398
Epoch 171/500
 - 0s - loss: 0.0552 - val_loss: 0.0394
Epoch 172/500
 - 0s - loss: 0.0547 - val_loss: 0.0389
Epoch 173/500
 - 0s - loss: 0.0542 - val_loss: 0.0384
Epoch 174/500
 - 0s - loss: 0.0538 - val_loss: 0.0380
Epoch 175/500
 - 0s - loss: 0.0533 - val_loss: 0.0375
Epoch 176/500
 - 0s - loss: 0.0529 - val_loss: 0.0372
Epoch 177/500
 - 0s - loss: 0.0525 - val_loss: 0.0367
Epoch 178/500
 - 0s - loss: 0.0520 - val_loss: 0.0362
Epoch 179/500
 - 0s - loss: 0.0516 - val_loss: 0.0359
Epoch 180/500
 - 0s - loss: 0.0512 - val_loss: 0.0354
Epoch 181/500
 - 0s - loss: 0.0508 - val_loss: 0.0350
Epoch 182/500
 - 0s - loss: 0.0504 - val_loss: 0.0348
Epoch 183/500
 - 0s - loss: 0.0500 - val_loss: 0.0342
Epoch 184/500
 - 0s - loss: 0.0496 - val_loss: 0.0339
Epoch 185/500
 - 0s - loss: 0.0492 - val_loss: 0.0335
Epoch 186/500
 - 0s - loss: 0.0488 - val_loss: 0.0331
Epoch 187/500
 - 0s - loss: 0.0484 - val_loss: 0.0327
Epoch 188/500
 - 0s - loss: 0.0480 - val_loss: 0.0324
Epoch 189/500
 - 0s - loss: 0.0477 - val_loss: 0.0320
Epoch 190/500
 - 0s - loss: 0.0473 - val_loss: 0.0316
Epoch 191/500
 - 0s - loss: 0.0469 - val_loss: 0.0314
Epoch 192/500
 - 0s - loss: 0.0466 - val_loss: 0.0309
Epoch 193/500
 - 0s - loss: 0.0462 - val_loss: 0.0307
Epoch 194/500
 - 0s - loss: 0.0459 - val_loss: 0.0303
Epoch 195/500
 - 0s - loss: 0.0456 - val_loss: 0.0300
Epoch 196/500
 - 0s - loss: 0.0452 - val_loss: 0.0296
Epoch 197/500
 - 0s - loss: 0.0449 - val_loss: 0.0293
Epoch 198/500
 - 0s - loss: 0.0446 - val_loss: 0.0291
Epoch 199/500
 - 0s - loss: 0.0443 - val_loss: 0.0287
Epoch 200/500
 - 0s - loss: 0.0439 - val_loss: 0.0285
Epoch 201/500
 - 0s - loss: 0.0436 - val_loss: 0.0282
Epoch 202/500
 - 0s - loss: 0.0433 - val_loss: 0.0278
Epoch 203/500
 - 0s - loss: 0.0430 - val_loss: 0.0276
Epoch 204/500
 - 0s - loss: 0.0427 - val_loss: 0.0273
Epoch 205/500
 - 0s - loss: 0.0424 - val_loss: 0.0270
Epoch 206/500
 - 0s - loss: 0.0422 - val_loss: 0.0268
Epoch 207/500
 - 0s - loss: 0.0419 - val_loss: 0.0265
Epoch 208/500
 - 0s - loss: 0.0416 - val_loss: 0.0262
Epoch 209/500
 - 0s - loss: 0.0413 - val_loss: 0.0260
Epoch 210/500
 - 0s - loss: 0.0410 - val_loss: 0.0258
Epoch 211/500
 - 0s - loss: 0.0408 - val_loss: 0.0255
Epoch 212/500
 - 0s - loss: 0.0405 - val_loss: 0.0252
Epoch 213/500
 - 0s - loss: 0.0403 - val_loss: 0.0250
Epoch 214/500
 - 0s - loss: 0.0400 - val_loss: 0.0248
Epoch 215/500
 - 0s - loss: 0.0398 - val_loss: 0.0246
Epoch 216/500
 - 0s - loss: 0.0395 - val_loss: 0.0243
Epoch 217/500
 - 0s - loss: 0.0393 - val_loss: 0.0241
Epoch 218/500
 - 0s - loss: 0.0390 - val_loss: 0.0239
Epoch 219/500
 - 0s - loss: 0.0388 - val_loss: 0.0236
Epoch 220/500
 - 0s - loss: 0.0385 - val_loss: 0.0234
Epoch 221/500
 - 0s - loss: 0.0383 - val_loss: 0.0233
Epoch 222/500
 - 0s - loss: 0.0381 - val_loss: 0.0230
Epoch 223/500
 - 0s - loss: 0.0379 - val_loss: 0.0228
Epoch 224/500
 - 0s - loss: 0.0376 - val_loss: 0.0226
Epoch 225/500
 - 0s - loss: 0.0374 - val_loss: 0.0224
Epoch 226/500
 - 0s - loss: 0.0372 - val_loss: 0.0223
Epoch 227/500
 - 0s - loss: 0.0370 - val_loss: 0.0220
Epoch 228/500
 - 0s - loss: 0.0368 - val_loss: 0.0218
Epoch 229/500
 - 0s - loss: 0.0366 - val_loss: 0.0217
Epoch 230/500
 - 0s - loss: 0.0364 - val_loss: 0.0215
Epoch 231/500
 - 0s - loss: 0.0362 - val_loss: 0.0212
Epoch 232/500
 - 0s - loss: 0.0360 - val_loss: 0.0212
Epoch 233/500
 - 0s - loss: 0.0358 - val_loss: 0.0210
Epoch 234/500
 - 0s - loss: 0.0357 - val_loss: 0.0207
Epoch 235/500
 - 0s - loss: 0.0355 - val_loss: 0.0207
Epoch 236/500
 - 0s - loss: 0.0353 - val_loss: 0.0205
Epoch 237/500
 - 0s - loss: 0.0351 - val_loss: 0.0201
Epoch 238/500
 - 0s - loss: 0.0349 - val_loss: 0.0202
Epoch 239/500
 - 0s - loss: 0.0348 - val_loss: 0.0200
Epoch 240/500
 - 0s - loss: 0.0346 - val_loss: 0.0198
Epoch 241/500
 - 0s - loss: 0.0344 - val_loss: 0.0198
Epoch 242/500
 - 0s - loss: 0.0343 - val_loss: 0.0195
Epoch 243/500
 - 0s - loss: 0.0341 - val_loss: 0.0194
Epoch 244/500
 - 0s - loss: 0.0339 - val_loss: 0.0193
Epoch 245/500
 - 0s - loss: 0.0338 - val_loss: 0.0191
Epoch 246/500
 - 0s - loss: 0.0336 - val_loss: 0.0190
Epoch 247/500
 - 0s - loss: 0.0335 - val_loss: 0.0189
Epoch 248/500
 - 0s - loss: 0.0333 - val_loss: 0.0188
Epoch 249/500
 - 0s - loss: 0.0332 - val_loss: 0.0187
Epoch 250/500
 - 0s - loss: 0.0330 - val_loss: 0.0185
Epoch 251/500
 - 0s - loss: 0.0329 - val_loss: 0.0184
Epoch 252/500
 - 0s - loss: 0.0328 - val_loss: 0.0183
Epoch 253/500
 - 0s - loss: 0.0326 - val_loss: 0.0182
Epoch 254/500
 - 0s - loss: 0.0325 - val_loss: 0.0181
Epoch 255/500
 - 0s - loss: 0.0324 - val_loss: 0.0180
Epoch 256/500
 - 0s - loss: 0.0322 - val_loss: 0.0178
Epoch 257/500
 - 0s - loss: 0.0321 - val_loss: 0.0178
Epoch 258/500
 - 0s - loss: 0.0320 - val_loss: 0.0177
Epoch 259/500
 - 0s - loss: 0.0319 - val_loss: 0.0174
Epoch 260/500
 - 0s - loss: 0.0317 - val_loss: 0.0174
Epoch 261/500
 - 0s - loss: 0.0316 - val_loss: 0.0173
Epoch 262/500
 - 0s - loss: 0.0315 - val_loss: 0.0172
Epoch 263/500
 - 0s - loss: 0.0314 - val_loss: 0.0171
Epoch 264/500
 - 0s - loss: 0.0313 - val_loss: 0.0169
Epoch 265/500
 - 0s - loss: 0.0312 - val_loss: 0.0169
Epoch 266/500
 - 0s - loss: 0.0311 - val_loss: 0.0168
Epoch 267/500
 - 0s - loss: 0.0310 - val_loss: 0.0167
Epoch 268/500
 - 0s - loss: 0.0308 - val_loss: 0.0166
Epoch 269/500
 - 0s - loss: 0.0307 - val_loss: 0.0165
Epoch 270/500
 - 0s - loss: 0.0306 - val_loss: 0.0164
Epoch 271/500
 - 0s - loss: 0.0305 - val_loss: 0.0164
Epoch 272/500
 - 0s - loss: 0.0305 - val_loss: 0.0162
Epoch 273/500
 - 0s - loss: 0.0304 - val_loss: 0.0162
Epoch 274/500
 - 0s - loss: 0.0303 - val_loss: 0.0161
Epoch 275/500
 - 0s - loss: 0.0302 - val_loss: 0.0160
Epoch 276/500
 - 0s - loss: 0.0301 - val_loss: 0.0160
Epoch 277/500
 - 0s - loss: 0.0300 - val_loss: 0.0159
Epoch 278/500
 - 0s - loss: 0.0299 - val_loss: 0.0158
Epoch 279/500
 - 0s - loss: 0.0298 - val_loss: 0.0157
Epoch 280/500
 - 0s - loss: 0.0297 - val_loss: 0.0157
Epoch 281/500
 - 0s - loss: 0.0296 - val_loss: 0.0156
Epoch 282/500
 - 0s - loss: 0.0296 - val_loss: 0.0156
Epoch 283/500
 - 0s - loss: 0.0295 - val_loss: 0.0154
Epoch 284/500
 - 0s - loss: 0.0294 - val_loss: 0.0154
Epoch 285/500
 - 0s - loss: 0.0293 - val_loss: 0.0153
Epoch 286/500
 - 0s - loss: 0.0293 - val_loss: 0.0153
Epoch 287/500
 - 0s - loss: 0.0292 - val_loss: 0.0153
Epoch 288/500
 - 0s - loss: 0.0291 - val_loss: 0.0151
Epoch 289/500
 - 0s - loss: 0.0290 - val_loss: 0.0151
Epoch 290/500
 - 0s - loss: 0.0290 - val_loss: 0.0150
Epoch 291/500
 - 0s - loss: 0.0289 - val_loss: 0.0149
Epoch 292/500
 - 0s - loss: 0.0288 - val_loss: 0.0150
Epoch 293/500
 - 0s - loss: 0.0288 - val_loss: 0.0148
Epoch 294/500
 - 0s - loss: 0.0287 - val_loss: 0.0147
Epoch 295/500
 - 0s - loss: 0.0286 - val_loss: 0.0147
Epoch 296/500
 - 0s - loss: 0.0286 - val_loss: 0.0147
Epoch 297/500
 - 0s - loss: 0.0285 - val_loss: 0.0147
Epoch 298/500
 - 0s - loss: 0.0285 - val_loss: 0.0146
Epoch 299/500
 - 0s - loss: 0.0284 - val_loss: 0.0147
Epoch 300/500
 - 0s - loss: 0.0284 - val_loss: 0.0144
Epoch 301/500
 - 0s - loss: 0.0283 - val_loss: 0.0144
Epoch 302/500
 - 0s - loss: 0.0282 - val_loss: 0.0145
Epoch 303/500
 - 0s - loss: 0.0282 - val_loss: 0.0143
Epoch 304/500
 - 0s - loss: 0.0281 - val_loss: 0.0144
Epoch 305/500
 - 0s - loss: 0.0281 - val_loss: 0.0142
Epoch 306/500
 - 0s - loss: 0.0280 - val_loss: 0.0143
Epoch 307/500
 - 0s - loss: 0.0280 - val_loss: 0.0144
Epoch 308/500
 - 0s - loss: 0.0279 - val_loss: 0.0141
Epoch 309/500
 - 0s - loss: 0.0279 - val_loss: 0.0142
Epoch 310/500
 - 0s - loss: 0.0278 - val_loss: 0.0140
Epoch 311/500
 - 0s - loss: 0.0278 - val_loss: 0.0140
Epoch 312/500
 - 0s - loss: 0.0277 - val_loss: 0.0142
Epoch 313/500
 - 0s - loss: 0.0277 - val_loss: 0.0142
Epoch 314/500
 - 0s - loss: 0.0277 - val_loss: 0.0141
Epoch 315/500
 - 0s - loss: 0.0276 - val_loss: 0.0141
Epoch 316/500
 - 0s - loss: 0.0276 - val_loss: 0.0139
Epoch 317/500
 - 0s - loss: 0.0275 - val_loss: 0.0139
Epoch 318/500
 - 0s - loss: 0.0275 - val_loss: 0.0140
Epoch 319/500
 - 0s - loss: 0.0275 - val_loss: 0.0139
Epoch 320/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 321/500
 - 0s - loss: 0.0274 - val_loss: 0.0139
Epoch 322/500
 - 0s - loss: 0.0273 - val_loss: 0.0139
Epoch 323/500
 - 0s - loss: 0.0273 - val_loss: 0.0139
Epoch 324/500
 - 0s - loss: 0.0273 - val_loss: 0.0138
Epoch 325/500
 - 0s - loss: 0.0272 - val_loss: 0.0138
Epoch 326/500
 - 0s - loss: 0.0272 - val_loss: 0.0138
Epoch 327/500
 - 0s - loss: 0.0272 - val_loss: 0.0137
Epoch 328/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 329/500
 - 0s - loss: 0.0271 - val_loss: 0.0137
Epoch 330/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 331/500
 - 0s - loss: 0.0270 - val_loss: 0.0135
Epoch 332/500
 - 0s - loss: 0.0270 - val_loss: 0.0137
Epoch 333/500
 - 0s - loss: 0.0270 - val_loss: 0.0135
Epoch 334/500
 - 0s - loss: 0.0270 - val_loss: 0.0136
Epoch 335/500
 - 0s - loss: 0.0269 - val_loss: 0.0134
Epoch 336/500
 - 0s - loss: 0.0269 - val_loss: 0.0136
Epoch 337/500
 - 0s - loss: 0.0269 - val_loss: 0.0133
Epoch 338/500
 - 0s - loss: 0.0269 - val_loss: 0.0136
Epoch 339/500
 - 0s - loss: 0.0268 - val_loss: 0.0134
Epoch 340/500
 - 0s - loss: 0.0268 - val_loss: 0.0132
Epoch 341/500
 - 0s - loss: 0.0268 - val_loss: 0.0132
Epoch 342/500
 - 0s - loss: 0.0268 - val_loss: 0.0133
Epoch 343/500
 - 0s - loss: 0.0267 - val_loss: 0.0133
Epoch 344/500
 - 0s - loss: 0.0267 - val_loss: 0.0132
Epoch 345/500
 - 0s - loss: 0.0267 - val_loss: 0.0133
Epoch 346/500
 - 0s - loss: 0.0267 - val_loss: 0.0130
Epoch 347/500
 - 0s - loss: 0.0266 - val_loss: 0.0132
Epoch 348/500
 - 0s - loss: 0.0266 - val_loss: 0.0131
Epoch 349/500
 - 0s - loss: 0.0266 - val_loss: 0.0132
Epoch 350/500
 - 0s - loss: 0.0266 - val_loss: 0.0132
Epoch 351/500
 - 0s - loss: 0.0266 - val_loss: 0.0132
Epoch 352/500
 - 0s - loss: 0.0265 - val_loss: 0.0131
Epoch 353/500
 - 0s - loss: 0.0265 - val_loss: 0.0130
Epoch 354/500
 - 0s - loss: 0.0265 - val_loss: 0.0131
Epoch 355/500
 - 0s - loss: 0.0265 - val_loss: 0.0131
Epoch 356/500
 - 0s - loss: 0.0265 - val_loss: 0.0130
Epoch 357/500
 - 0s - loss: 0.0265 - val_loss: 0.0129
Epoch 358/500
 - 0s - loss: 0.0264 - val_loss: 0.0130
Epoch 359/500
 - 0s - loss: 0.0264 - val_loss: 0.0129
Epoch 360/500
 - 0s - loss: 0.0264 - val_loss: 0.0129
Epoch 361/500
 - 0s - loss: 0.0264 - val_loss: 0.0128
Epoch 362/500
 - 0s - loss: 0.0264 - val_loss: 0.0128
Epoch 363/500
 - 0s - loss: 0.0264 - val_loss: 0.0127
Epoch 364/500
 - 0s - loss: 0.0264 - val_loss: 0.0129
Epoch 365/500
 - 0s - loss: 0.0263 - val_loss: 0.0127
Epoch 366/500
 - 0s - loss: 0.0263 - val_loss: 0.0128
Epoch 367/500
 - 0s - loss: 0.0263 - val_loss: 0.0128
Epoch 368/500
 - 0s - loss: 0.0263 - val_loss: 0.0128
Epoch 369/500
 - 0s - loss: 0.0263 - val_loss: 0.0128
Epoch 370/500
 - 0s - loss: 0.0263 - val_loss: 0.0127
Epoch 371/500
 - 0s - loss: 0.0263 - val_loss: 0.0129
Epoch 372/500
 - 0s - loss: 0.0263 - val_loss: 0.0126
Epoch 373/500
 - 0s - loss: 0.0262 - val_loss: 0.0129
Epoch 374/500
 - 0s - loss: 0.0262 - val_loss: 0.0126
Epoch 375/500
 - 0s - loss: 0.0262 - val_loss: 0.0128
Epoch 376/500
 - 0s - loss: 0.0262 - val_loss: 0.0127
Epoch 377/500
 - 0s - loss: 0.0262 - val_loss: 0.0127
Epoch 378/500
 - 0s - loss: 0.0262 - val_loss: 0.0127
Epoch 379/500
 - 0s - loss: 0.0262 - val_loss: 0.0126
Epoch 380/500
 - 0s - loss: 0.0262 - val_loss: 0.0126
Epoch 381/500
 - 0s - loss: 0.0262 - val_loss: 0.0125
Epoch 382/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 383/500
 - 0s - loss: 0.0261 - val_loss: 0.0124
Epoch 384/500
 - 0s - loss: 0.0261 - val_loss: 0.0127
Epoch 385/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 386/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 387/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 388/500
 - 0s - loss: 0.0261 - val_loss: 0.0126
Epoch 389/500
 - 0s - loss: 0.0261 - val_loss: 0.0124
Epoch 390/500
 - 0s - loss: 0.0261 - val_loss: 0.0125
Epoch 391/500
 - 0s - loss: 0.0261 - val_loss: 0.0124
Epoch 392/500
 - 0s - loss: 0.0260 - val_loss: 0.0125
Epoch 393/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 394/500
 - 0s - loss: 0.0260 - val_loss: 0.0125
Epoch 395/500
 - 0s - loss: 0.0260 - val_loss: 0.0126
Epoch 396/500
 - 0s - loss: 0.0260 - val_loss: 0.0125
Epoch 397/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 398/500
 - 0s - loss: 0.0260 - val_loss: 0.0125
Epoch 399/500
 - 0s - loss: 0.0260 - val_loss: 0.0123
Epoch 400/500
 - 0s - loss: 0.0260 - val_loss: 0.0126
Epoch 401/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 402/500
 - 0s - loss: 0.0260 - val_loss: 0.0126
Epoch 403/500
 - 0s - loss: 0.0260 - val_loss: 0.0125
Epoch 404/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 405/500
 - 0s - loss: 0.0260 - val_loss: 0.0124
Epoch 406/500
 - 0s - loss: 0.0260 - val_loss: 0.0123
Epoch 407/500
 - 0s - loss: 0.0259 - val_loss: 0.0125
Epoch 408/500
 - 0s - loss: 0.0259 - val_loss: 0.0123
Epoch 409/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 410/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 411/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 412/500
 - 0s - loss: 0.0259 - val_loss: 0.0123
Epoch 413/500
 - 0s - loss: 0.0259 - val_loss: 0.0125
Epoch 414/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 415/500
 - 0s - loss: 0.0259 - val_loss: 0.0125
Epoch 416/500
 - 0s - loss: 0.0259 - val_loss: 0.0123
Epoch 417/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 418/500
 - 0s - loss: 0.0259 - val_loss: 0.0123
Epoch 419/500
 - 0s - loss: 0.0259 - val_loss: 0.0121
Epoch 420/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 421/500
 - 0s - loss: 0.0259 - val_loss: 0.0121
Epoch 422/500
 - 0s - loss: 0.0258 - val_loss: 0.0124
Epoch 423/500
 - 0s - loss: 0.0259 - val_loss: 0.0122
Epoch 424/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 425/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 426/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 427/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 428/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 429/500
 - 0s - loss: 0.0259 - val_loss: 0.0121
Epoch 430/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 431/500
 - 0s - loss: 0.0258 - val_loss: 0.0119
Epoch 432/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 433/500
 - 0s - loss: 0.0258 - val_loss: 0.0120
Epoch 434/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 435/500
 - 0s - loss: 0.0258 - val_loss: 0.0121
Epoch 436/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 437/500
 - 0s - loss: 0.0258 - val_loss: 0.0121
Epoch 438/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 439/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 440/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 441/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 442/500
 - 0s - loss: 0.0258 - val_loss: 0.0121
Epoch 443/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 444/500
 - 0s - loss: 0.0258 - val_loss: 0.0120
Epoch 445/500
 - 0s - loss: 0.0257 - val_loss: 0.0123
Epoch 446/500
 - 0s - loss: 0.0258 - val_loss: 0.0119
Epoch 447/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 448/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 449/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 450/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 451/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 452/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 453/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 454/500
 - 0s - loss: 0.0258 - val_loss: 0.0121
Epoch 455/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 456/500
 - 0s - loss: 0.0258 - val_loss: 0.0119
Epoch 457/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 458/500
 - 0s - loss: 0.0257 - val_loss: 0.0119
Epoch 459/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 460/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 461/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 462/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 463/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 464/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 465/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 466/500
 - 0s - loss: 0.0258 - val_loss: 0.0119
Epoch 467/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 468/500
 - 0s - loss: 0.0257 - val_loss: 0.0119
Epoch 469/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 470/500
 - 0s - loss: 0.0257 - val_loss: 0.0119
Epoch 471/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 472/500
 - 0s - loss: 0.0257 - val_loss: 0.0119
Epoch 473/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 474/500
 - 0s - loss: 0.0257 - val_loss: 0.0119
Epoch 475/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 476/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 477/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 478/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 479/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 480/500
 - 0s - loss: 0.0257 - val_loss: 0.0119
Epoch 481/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 482/500
 - 0s - loss: 0.0257 - val_loss: 0.0118
Epoch 483/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 484/500
 - 0s - loss: 0.0256 - val_loss: 0.0118
Epoch 485/500
 - 0s - loss: 0.0256 - val_loss: 0.0119
Epoch 486/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 487/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 488/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 489/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 490/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 491/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 492/500
 - 0s - loss: 0.0257 - val_loss: 0.0118
Epoch 493/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 494/500
 - 0s - loss: 0.0257 - val_loss: 0.0117
Epoch 495/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 496/500
 - 0s - loss: 0.0257 - val_loss: 0.0117
Epoch 497/500
 - 0s - loss: 0.0256 - val_loss: 0.0122
Epoch 498/500
 - 0s - loss: 0.0256 - val_loss: 0.0117
Epoch 499/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 500/500
 - 0s - loss: 0.0256 - val_loss: 0.0117
In [88]:
pyplot.plot(history['loss'], label='train')
pyplot.plot(history['val_loss'], label='validation')
pyplot.legend()
pyplot.show()

Test improved model on Validation Data

In [89]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(validation_X,validation_y,model,scaler)
print('LSTM Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Model on Validation Data RMSE: 8.984

Test improved model on Unseen(Test) Data

In [90]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(test_X,test_y,model,scaler)
print('LSTM Moddel on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Moddel on Test Data RMSE: 8.709